jsonata custom manager has different behavior than regex #34567
Replies: 6 comments 22 replies
-
|
Hi there, Please help this Discussion progress by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible. Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this. Discussions without reproductions are less likely to be converted to Issues. Please follow these steps:
If you need help with running renovate on your minimal reproduction repository, please refer to our Running Renovate guide. The Renovate team |
Beta Was this translation helpful? Give feedback.
-
|
It seems like a valid bug to me as I didn't test the implementation of jsonata manager on a digest. Please, provide a minimal reproduction and I will look into it. |
Beta Was this translation helpful? Give feedback.
-
|
I have added a minimal reproduction: https://github.com/coffee2go/renovate-34567 Do I need to give you write access to the repo? |
Beta Was this translation helpful? Give feedback.
-
|
More logs: DEBUG: packageFiles with updates (repository=coffee2go/renovate-34567, baseBranch=main)
"config": {
"jsonata": [
{
"deps": [
{
"depName": "nginx",
"currentValue": "1.26.3",
"currentDigest": "sha256:73d149c5b564e7a243060ae939b7c789f083ca8d9131e16f90246e596948d858",
"datasource": "docker",
"updates": [
{
"bucket": "non-major",
"newVersion": "1.27.4",
"newValue": "1.27.4",
"newDigest": "sha256:9d6b58feebd2dbd3c56ab5853333d627cc6e281011cfd6050fa4bcf2072c9496",
"releaseTimestamp": "2025-02-25T06:51:45.576Z",
"newVersionAgeInDays": 4,
"newMajor": 1,
"newMinor": 27,
"newPatch": 4,
"updateType": "minor",
"libYears": 0.0000011302955352612887,
"branchName": "renovate/nginx-1.x"
},
{
"updateType": "digest",
"newValue": "1.26.3",
"newDigest": "sha256:be4f49e87cc022c1332bd4a9d84fb66bd76c72b6399b57193adec00ec9a616d2",
"branchName": "renovate/nginx-1.26.3"
}
],
"packageName": "nginx",
"versioning": "docker",
"warnings": [],
"registryUrl": "https://index.docker.io",
"lookupName": "library/nginx",
"currentVersion": "1.26.3",
"currentVersionTimestamp": "2025-02-25T06:51:09.931Z",
"currentVersionAgeInDays": 4,
"isSingleVersion": true,
"fixedVersion": "1.26.3"
}
],
"matchStrings": [
"[ { \"depName\": image.name, \"currentValue\": image.tag, \"currentDigest\": image.digest } ]"
],
"fileFormat": "yaml",
"datasourceTemplate": "docker",
"packageFile": "values.yaml"
},
{
"deps": [
{
"depName": "nginx",
"currentValue": "1.26.3",
"currentDigest": "sha256:73d149c5b564e7a243060ae939b7c789f083ca8d9131e16f90246e596948d858",
"datasource": "docker",
"updates": [
{
"bucket": "non-major",
"newVersion": "1.27.4",
"newValue": "1.27.4",
"newDigest": "sha256:9d6b58feebd2dbd3c56ab5853333d627cc6e281011cfd6050fa4bcf2072c9496",
"releaseTimestamp": "2025-02-25T06:51:45.576Z",
"newVersionAgeInDays": 4,
"newMajor": 1,
"newMinor": 27,
"newPatch": 4,
"updateType": "minor",
"libYears": 0.0000011302955352612887,
"branchName": "renovate/nginx-1.x"
},
{
"updateType": "digest",
"newValue": "1.26.3",
"newDigest": "sha256:be4f49e87cc022c1332bd4a9d84fb66bd76c72b6399b57193adec00ec9a616d2",
"branchName": "renovate/nginx-1.26.3"
}
],
"packageName": "nginx",
"versioning": "docker",
"warnings": [],
"registryUrl": "https://index.docker.io",
"lookupName": "library/nginx",
"currentVersion": "1.26.3",
"currentVersionTimestamp": "2025-02-25T06:51:09.931Z",
"currentVersionAgeInDays": 4,
"isSingleVersion": true,
"fixedVersion": "1.26.3"
}
],
"matchStrings": [
"[ { \"depName\": image.name, \"currentValue\": image.tag, \"currentDigest\": image.digest } ]"
],
"fileFormat": "yaml",
"datasourceTemplate": "docker",
"packageFile": "values.yaml"
}
]
}
|
Beta Was this translation helpful? Give feedback.
-
|
No write access needed - we can fork |
Beta Was this translation helpful? Give feedback.
-
|
I have the same problem. One thing I noticed while trying to debug the issue is that the renovate/lib/modules/manager/custom/regex/index.ts Lines 67 to 69 in 025d0fd It seems that those lines are missing from the JSONata manager, meaning that the key is simply ignored no matter what. |
Beta Was this translation helpful? Give feedback.
-
|
replacing string is not extracted as we do not match the exact string in jsonata manager, we transform the original file content to json object hence extact matching in not possible. |
Beta Was this translation helpful? Give feedback.
-
|
I am also experiencing this error and have the same error messages. Self-hosted Renovate, Gitlab, renovate version 39.238.1 My config and source file ( {
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"customManagers": [
{
"customType": "jsonata",
"fileFormat": "yaml",
"fileMatch": ["images/.*.yaml"],
"datasourceTemplate": "docker",
"matchStrings": [
"spec.sources.{ \"packageName\": registry & \"/\" & repository, \"currentValue\": tag, \"currentDigest\": digest }"
]
}
]
}---
spec:
sources:
- registry: docker.io
repository: hashicorp/vault
tag: "1.18.2"
digest: "sha256:2090eb7ac7a4bdef802f685698bd4dc0740de683affe8ff7df55f4fc77077ba7" |
Beta Was this translation helpful? Give feedback.
-
|
Will be looking into it, next week. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I'd also like to add my case. My use case is a custom helm chart in Helmfile, which is a simple wrapper for web apps. The regex manager correctly works, while the JSONata manager gives the following errors. releases:
- name: app1
chart: ../../charts/single-app
namespace: apps
values:
- image:
name: example.com/app1
tag: "latest@sha256:d62c46bec018305a5a2bf34219899f06bc271470be03204cb2dae8ae18709b95" "customManagers": [
{
"customType": "regex",
"managerFilePatterns": [
"/(^|/)(.*\\.)?helmfile\\.ya?ml(?:\\.gotmpl)?$/"
],
"matchStrings": [
"\\n - name: \"?(?<depName>[^\"\\s]+)\"?\\n(?:[^-]|\\n)*?chart: .*single-app\"?\\n(?:.|\\n)*?- image:\\n(?:.|\\n)*?name: \"?(?<packageName>[^\"\\s]+)\"?\\n(?:.|\\n)*?tag: \"?(?<currentValue>[^@\"\\s]+)(?:@(?<currentDigest>sha256:[^@\"\\s]+))?\"?"
],
"datasourceTemplate": "docker"
},
{
"customType": "jsonata",
"managerFilePatterns": [
"/(^|/)(.*\\.)?helmfile\\.ya?ml(?:\\.gotmpl)?$/"
],
"fileFormat": "yaml",
"matchStrings": [
"$map($filter(releases, function($v){$contains($v.chart, \"charts/single-app\")}), function($v){{\n \"depName\": $v.name & \"-jsonata\",\n \"packageName\": $v.values.image.name,\n \"currentValue\": $split($v.values.image.tag, \"@\")[0],\n \"currentDigest\": $split($v.values.image.tag, \"@\")[1]\n}})"
],
"datasourceTemplate": "docker"
}
], |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I've simplified the demo. I've kept the different PRs to show that the regex manager works correctly and that JSONata also works correctly with a non-digest version. If you wish I can also remove them all. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Here is the log with the new repo: Is that part correct so far? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, exactly, that's my experience too. Parsing is correctly done on JSONata renovate-sha and new digest is correctly fetched. It's when the engine tries to replace the string that it fails. |
Beta Was this translation helpful? Give feedback.
-
I tried this as starter and it works: diff --git a/lib/workers/repository/update/branch/auto-replace.ts b/lib/workers/repository/update/branch/auto-replace.ts
index af70427c4..2b40b0043 100644
--- a/lib/workers/repository/update/branch/auto-replace.ts
+++ b/lib/workers/repository/update/branch/auto-replace.ts
@@ -225,10 +225,11 @@ export async function doAutoReplace(
return await checkExistingBranch(upgrade, existingContent);
}
const replaceWithoutReplaceString =
- is.string(newName) &&
- newName !== depName &&
- (is.undefined(upgrade.replaceString) ||
- !upgrade.replaceString?.includes(depName!));
+ upgrade.manager === 'jsonata' ||
+ (is.string(newName) &&
+ newName !== depName &&
+ (is.undefined(upgrade.replaceString) ||
+ !upgrade.replaceString?.includes(depName!)));
const replaceString = upgrade.replaceString ?? currentValue ?? currentDigest;
logger.trace({ depName, replaceString }, 'autoReplace replaceString');
let searchIndex: number;
@@ -377,6 +378,15 @@ export async function doAutoReplace(
currentValue!,
newValue,
);
+ if (newDigest) {
+ const digestIndex = newContent.indexOf(currentDigest!, searchIndex);
+ newContent = replaceAt(
+ newContent,
+ digestIndex,
+ currentDigest!,
+ newDigest,
+ );
+ }
await writeLocalFile(upgrade.packageFile!, newContent);
valueReplaced = true;
searchIndex += newValue.length - 1; |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
I run into this issue as well and would like to know, if there I noticed that JSONata manager works fine for me, if the version has 3 parts, but not, if it has 4 parts. We use custom JSON files in our repository, which provides us good flexibility. We import those JSON files in CDK Code for example. |
Beta Was this translation helpful? Give feedback.
-
|
I want to clarify that The Mar 20 2025 comment by @RahulGautamSingh ("exact matching is not possible") is correct for the Concrete example: Business Central AL extensionsSetup: Self-hosted Renovate (GitHub Actions), GitHub, Renovate v43.150.0 Microsoft Dynamics 365 Business Central stores dependency versions in a mandatory 4-part format ( app.json: {
"id": "a8f9e1d2-3b4c-5678-9abc-def012345678",
"name": "AL Example Renovate",
"publisher": "Example Publisher",
"version": "1.0.0.0",
"application": "25.0.0.0",
"platform": "25.0.0.0",
"dependencies": [
{
"id": "63ca2fa4-4f03-4f2b-a480-172fef340d3f",
"name": "System Application",
"publisher": "Microsoft",
"version": "25.0.0.0"
},
{
"id": "5491b2df-228f-423d-b36a-bcfb380f4f5a",
"name": "1Vision Moxa-2-PrintVis IoT Connector",
"publisher": "1Vision365",
"version": "26.0.0.1"
}
]
}Renovate config {
"customManagers": [
{
"customType": "jsonata",
"fileFormat": "json",
"managerFilePatterns": ["**/app.json"],
"matchStrings": [
"dependencies[publisher!='Microsoft'].{\"depName\": publisher & '/' & name, \"packageName\": $replace(publisher, /\\s+/, '') & '.' & $replace(name, /\\s+/, '') & '.symbols.' & id, \"currentValue\": version}"
],
"datasourceTemplate": "nuget",
"registryUrlTemplate": "https://pkgs.dev.azure.com/dynamicssmb2/DynamicsBCPublicFeeds/_packaging/AppSourceSymbols/nuget/v3/index.json",
"versioningTemplate": "nuget",
"autoReplaceStringTemplate": "{{replace \"^(\\\\d+\\\\.\\\\d+\\\\.\\\\d+)$\" \"$1.0\" newValue}}"
}
]
}The What happens Renovate detects that - "version": "26.0.0.1"
+ "version": "27.0.0"The BC compiler rejects this. The correct write-back is Why this is not covered by the digest fixThe Jun 6 fix makes For Workaround (self-hosted only)
"postUpgradeTasks": {
"commands": [
"sed -i -E 's/(\"(version|application|platform)\": \")([0-9]+\\.[0-9]+\\.[0-9]+)\"/\\1\\3.0\"/g' {{{packageFile}}}"
],
"fileFilters": ["**/app.json"],
"executionMode": "update"
}This requires |
Beta Was this translation helpful? Give feedback.
-
|
@RahulGautamSingh Thanks for the pointer. I tested PR #42840 against the BC use case and it does not resolve 1. Propagation is still missing PR #42840 only touches 2. Even with propagation fixed, This is the deeper issue. After The autoreplace mechanism implicitly assumes the template output equals A complete fix would require one or both of:
In the meantime, |
Beta Was this translation helpful? Give feedback.
-
|
I create this new PR to handle that issue: #43535 Can you test it and confirm the fix is working for you? |
Beta Was this translation helpful? Give feedback.
-
|
@RahulGautamSingh sorry for the late reply. I tested with your branch but wasn't able to successfully apply the template. It still proposed to update the app to 27.0.0 instead of replacing it with 27.0.0.0 via the template |
Beta Was this translation helpful? Give feedback.
-
|
The regex you're using is just double-escaped. In JSON, Use single escaping: (\d, not \\d) "autoReplaceStringTemplate": "{{replace \"^(\\d+\\.\\d+\\.\\d+)$\" \"$1.0\" newValue}}" |
Beta Was this translation helpful? Give feedback.
-
|
@RahulGautamSingh My bad, sorry. But with the correct escaping it still doesn't work unfortunately. I have added the debug log to this comment. The repo I tried it on is https://github.com/axiansinfoma/al-example-renovate/tree/test-template-replace and the dependency is {
"id": "5491b2df-228f-423d-b36a-bcfb380f4f5a",
"name": "1Vision Moxa-2-PrintVis IoT Connector",
"publisher": "1Vision365",
"version": "26.0.0.1" // latest available is 27.0.0.0
} |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
How are you running Renovate?
Self-hosted Renovate
If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.
bitbucket, renovate 39.185.0
Please tell us more about your question or problem
When running renovate with this custom manager, everything works:
Notice that the capture groups are
depName,currentDigestandcurrentValue. The result is expected and the values are replaced accordingly. Bothtaganddigestare updated in this case. However when using the seemingly equivalent (but more robust) jsonata manager, it breaks:I am getting log messages like
See logs for more. (I've also tried returning both an array and non-array in matchStrings, which didn't change anything, I think it made it worse).
Logs (if relevant)
Logs
Beta Was this translation helpful? Give feedback.
All reactions