Skip to content

Commit 33ca449

Browse files
elaine-mattosqtomlinsonjeffmendoza
authored
feat: update nodejs and other packages (#1340)
* feat: migrate node Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * feat: update octokit Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: add defensive checks Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: use promise-based mkdirp directly without promisify due to changes in the latest mkdip version Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: remove old express-init Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: encode deflated data as base64 for Redis compatibility Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: update ajv Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix(routes): update route definitions for express v5 compatibility - Added support for the /splash endpoint - Updated optional route parameters to Express v5 syntax ({/:param}) - Adjusted root path and wildcards for routes like /definitions and /harvest Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: update express Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: update yaml Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: update gitbeaker - Removed defaultHeaders due to incompatibility with new version. See: jdalrymple/gitbeaker#3634 Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * chore: improve winston config Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix(harvest): instantiate with caching again Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: update minimatch Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * chore: improve error handling Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * chore: push package updates Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix(tests): update octokit calls, fix deep equalities, initialize logger Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * refactor(logging): replace winston-azure-application-insights with native AppInsights SDK - Adds support for a new env variable to control console logging output. - The third-party transport has been removed in favor of a direct integration using the SDK. Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: linting issues Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix(redis): add type annotation to bypass TLS configuration type issue - The Redis client accepts as a valid runtime configuration, but the TypeScript definitions expect a different structure. Added @ts-ignore directive to bypass the type check. Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: add back comment Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: update node version Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: eslint errors Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * chore: try and fix codeQL issues with ajv schema validation Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix(harvest): return structured JSON error responses for validation and input size Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * test(harvest): fix tests Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * test: add extra check suggested by copilot Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * chore: log strict mode issues Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: escape regex and limit pattern length to prevent ReDoS Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * refactor(logging): improve error reporting and sanitize internal error responses Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: validate and limit patch inputs to prevent deep object traversal Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: add security limits to the ajv instance Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: escape HTML to prevent XSS and improve response format with res.json() Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * feat: add input prevalidation middleware to guard against large or deeply nested JSON bodies Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: escape user input in RegExp to prevent ReDoS and ensure safe coordinate queries Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: prettier Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: make allErrors dependent on config Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: update test script to enable REST_DEBUG during test runs Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: update to node 24 Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix(redis): make tls optional and ensure socket options match RedisSocketOptions type Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix(logging): replace placeholders with structured logging for better Winston compatibility and performance Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix(redis): default redis port from config or fallback to 6379 Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: format Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: log error Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: improve error logging Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: improve mapLevel with Map and update JSDoc types Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: remove node 18 backwards compatibility and downgrade mongodb Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: remove node 18 backwards compatibility and downgrade mongodb Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: replace deprecated LoggerInstance with Logger in factory declaration Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix(startup): add app.init hook and execute before listening Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: decrease request body size limit Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * feat: remove deprecated useNewUrlParser and add startup log message * feat: use appVersion from config * fix: remove duplicated line Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: prevent double appInsights.start and simplify echo flag Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix(originGitHub): update GitHub calls and add structured logging - Replace deprecated gitdata.getTags and gitdata.getTag with the new rest.repos.listTags and the rest.git.getTag - Replace console with structured logging - Handle annotated and lightweight tags Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * test(originGitHub): add unit tests for GitHub origin routes Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: remove inputPreValidation Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * refactor(abstractFileStore): simplify query filtering with optional chaining Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * test(abstractFileStore): expand test coverage for find method Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: add back only necessary changes to curation route Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: add back only necessary changes to definitions route Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * text(mongoDB): add logger.info to tests Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * test(definitions): add extra params to tests Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: return all error messages with send instead of json Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * test(harvestRoute): remove json parsing Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: remove isTooDeepOrLarge function Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * feat(cache): add backward compatibility for legacy Redis cache data Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix(tests): add appVersion back to the environment variable to prevent config from getting loaded during tests Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * feat: make ajv allErrors control variable more explicit Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix(logging): change event listener from info to data on Winston logger to correctly pipe logs to appInsights Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: simplify cache item retrieval by removing fallback mechanism Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix(webhook): handle body parsing for body-parser 2.x compatibility Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> * fix: remove test case for unused cache fallback strategy Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> --------- Signed-off-by: ElaineDeMattosSilvaB <elaine.de-mattos-silva-bezerra@deutschebahn.com> Co-authored-by: Qing Tomlinson <qing.tomlinson@sap.com> Co-authored-by: Jeff Mendoza <jlm@jlm.name>
1 parent fdaca07 commit 33ca449

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

69 files changed

+7456
-8200
lines changed

.github/workflows/test.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ jobs:
2323

2424
- uses: actions/setup-node@v6
2525
with:
26-
node-version: 18
26+
node-version: 24
2727
cache: 'npm'
2828
registry-url: 'https://npm.pkg.github.com'
2929
scope: '@clearlydefined'

DevDockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Copyright (c) Microsoft Corporation and others. Licensed under the MIT license.
22
# SPDX-License-Identifier: MIT
33

4-
FROM docker.io/library/node:18
4+
FROM docker.io/library/node:24-bullseye
55
ENV APPDIR=/opt/service
66

77
## get SSH server running

Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Copyright (c) Microsoft Corporation and others. Licensed under the MIT license.
22
# SPDX-License-Identifier: MIT
33

4-
FROM docker.io/library/node:18
4+
FROM docker.io/library/node:24-bullseye
55
ENV APPDIR=/opt/service
66

77
## get SSH server running

app.js

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -28,13 +28,8 @@ function createApp(config) {
2828

2929
const summaryService = require('./business/summarizer')(config.summary)
3030

31-
const cachingService = config.caching.service()
32-
initializers.push(async () => cachingService.initialize())
33-
3431
const harvestStore = config.harvest.store()
3532
initializers.push(async () => harvestStore.initialize())
36-
const harvestService = config.harvest.service({ cachingService })
37-
const harvestRoute = require('./routes/harvest')(harvestService, harvestStore, summaryService)
3833

3934
const aggregatorService = require('./business/aggregator')(config.aggregator)
4035

@@ -50,6 +45,12 @@ function createApp(config) {
5045
const searchService = config.search.service()
5146
initializers.push(async () => searchService.initialize())
5247

48+
const cachingService = config.caching.service()
49+
initializers.push(async () => cachingService.initialize())
50+
51+
const harvestService = config.harvest.service({ cachingService })
52+
const harvestRoute = require('./routes/harvest')(harvestService, harvestStore, summaryService)
53+
5354
const curationService = config.curation.service(null, curationStore, config.endpoints, cachingService, harvestStore)
5455

5556
const curationQueue = config.curation.queue()
@@ -109,9 +110,11 @@ function createApp(config) {
109110

110111
const app = express()
111112
app.use(cors())
112-
app.options('*', cors())
113+
// new express v5 matching syntax: https://expressjs.com/en/guide/migrating-5.html#path-syntax
114+
app.options('*splat', cors())
115+
app.use(express.json({ limit: '100kb' }))
113116
app.use(cookieParser())
114-
app.use(helmet())
117+
app.use(helmet.default())
115118
app.use(requestId())
116119
app.use('/schemas', express.static('./schemas'))
117120

@@ -199,7 +202,6 @@ function createApp(config) {
199202
url: req.url
200203
})
201204
const err = new Error('Not Found')
202-
// @ts-ignore - Express error convention
203205
err.status = 404
204206
next(err)
205207
}
@@ -229,6 +231,8 @@ function createApp(config) {
229231
)
230232
}
231233

234+
app.init = requestHandler.init
235+
232236
// error handler
233237
app.use((error, request, response, next) => {
234238
if (response.headersSent) return next(error)
@@ -250,7 +254,7 @@ function createApp(config) {
250254
error: {
251255
code: status.toString(),
252256
message: 'An error has occurred',
253-
innererror: serializeError(response.locals.error)
257+
innererror: serializeError.serializeError(response.locals.error)
254258
}
255259
})
256260
})

bin/config.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ function loadOne(spec, namespace) {
3232
if (!target) target = require(requirePath)
3333
return objectPath ? get(target, objectPath) : target
3434
} catch (e) {
35-
throw new Error(`could not load provider for ${requirePath}`)
35+
throw new Error(`could not load provider for ${requirePath}. Error ${e.message}`)
3636
}
3737
}
3838

bin/www

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@ const config = require('./config')
66
const app = require('../app')(config)
77
const debug = require('debug')('service:server')
88
const http = require('http')
9-
const init = require('express-init')
109

1110
/**
1211
* Get port from environment and store in Express.
@@ -19,19 +18,30 @@ app.set('port', port)
1918
*/
2019
const server = http.createServer(app)
2120

21+
startServer()
22+
2223
/**
2324
* Initialize the apps (if they have async init functions) and start listening
2425
*/
25-
init(app, error => {
26-
if (error) {
27-
console.log('Error initializing the Express app: ' + error)
28-
throw new Error(error)
26+
async function startServer() {
27+
try {
28+
if (typeof app.init === 'function') {
29+
await new Promise((resolve, reject) => {
30+
app.init(app, err => {
31+
if (err) reject(err)
32+
else resolve()
33+
})
34+
})
35+
}
36+
server.listen(port)
37+
server.on('error', onError)
38+
server.on('listening', onListening)
39+
console.log(`Service listening on port: ${port}`)
40+
} catch (error) {
41+
console.error('Error initializing the Express app:', error)
42+
process.exit(1)
2943
}
30-
server.listen(port)
31-
server.on('error', onError)
32-
server.on('listening', onListening)
33-
console.log(`Service listening on port: ${port}`)
34-
})
44+
}
3545

3646
/**
3747
* Normalize a port into a number, string, or false.
@@ -56,11 +66,9 @@ function onError(error) {
5666
case 'EACCES':
5767
console.error(bind + ' requires elevated privileges')
5868
process.exit(1)
59-
break
6069
case 'EADDRINUSE':
6170
console.error(bind + ' is already in use')
6271
process.exit(1)
63-
break
6472
default:
6573
throw error
6674
}

business/definitionService.js

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ const {
2626
simplifyAttributions,
2727
updateSourceLocation
2828
} = require('../lib/utils')
29-
const minimatch = require('minimatch')
29+
const { minimatch } = require('minimatch')
3030
const extend = require('extend')
3131
const logger = require('../providers/logging/logger')
3232
const validator = require('../schemas/validator')
@@ -74,6 +74,7 @@ class DefinitionService {
7474
const curation = this.curationService.get(coordinates, pr)
7575
return this.compute(coordinates, curation)
7676
}
77+
7778
const existing = await this._cacheExistingAside(coordinates, force)
7879
let result = await this.upgradeHandler.validate(existing)
7980
if (result) {
@@ -186,6 +187,9 @@ class DefinitionService {
186187
try {
187188
return await this.list(coordinates)
188189
} catch (error) {
190+
this.logger.error('failed to list definitions', {
191+
error: error.message
192+
})
189193
return null
190194
}
191195
})
@@ -309,9 +313,10 @@ class DefinitionService {
309313
})
310314
}
311315
}
312-
313316
async _store(definition) {
317+
this.logger.debug('storing definition', { coordinates: definition.coordinates.toString() })
314318
await this.definitionStore.store(definition)
319+
this.logger.debug('definition stored successfully', { coordinates: definition.coordinates.toString() })
315320
await this._setDefinitionInCache(this._getCacheKey(definition.coordinates), definition)
316321
await this.harvestService.done(definition.coordinates)
317322
}
@@ -458,12 +463,14 @@ class DefinitionService {
458463
if (!declared || !discovered) return 0
459464
return discovered.every(expression => SPDX.satisfies(expression, declared)) ? weights.consistency : 0
460465
}
461-
462466
_computeSPDXScore(definition) {
463467
try {
464468
parse(get(definition, 'licensed.declared')) // use strict spdx-expression-parse
465469
return weights.spdx
466470
} catch (e) {
471+
this.logger.debug('Could not parse declared license expression.', {
472+
errorMessage: e.message
473+
})
467474
return 0
468475
}
469476
}

full.env.json

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,5 +51,9 @@
5151

5252
"========== Heapstats Logging settings (OPTIONAL) ==========": "",
5353
"LOG_NODE_HEAPSTATS": "<true|false>",
54-
"LOG_NODE_HEAPSTATS_INTERVAL_MS": "<time_in_milliseconds (e.g. '30000')>"
54+
"LOG_NODE_HEAPSTATS_INTERVAL_MS": "<time_in_milliseconds (e.g. '30000')>",
55+
56+
"========== Logging settings ==========": "",
57+
"ENABLE_REST_VALIDATION_ERRORS": "<true|false>",
58+
"LOGGER_LOG_TO_CONSOLE": "<true|false>"
5559
}

lib/curation.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ class Curation {
2828

2929
load(content) {
3030
try {
31-
return yaml.safeLoad(content)
31+
return yaml.load(content)
3232
} catch (error) {
3333
this.errors.push({ message: 'Invalid yaml', error })
3434
}

lib/github.d.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
* GitHub API client instance from @octokit/rest package. This represents the authenticated GitHub client with all
66
* available API methods.
77
*/
8-
export type GitHubClient = import('@octokit/rest')
8+
export type GitHubClient = import('@octokit/rest').Octokit
99

1010
/** Options for configuring GitHub client authentication. */
1111
export interface GitHubClientOptions {

0 commit comments

Comments
 (0)