Skip to content

[fix][backend] populate FieldData key from dataset schema in data reflow#562

Merged
SharkeyChen merged 2 commits into
mainfrom
fix/fielddata-empty-key-nil
Jun 24, 2026
Merged

[fix][backend] populate FieldData key from dataset schema in data reflow#562
SharkeyChen merged 2 commits into
mainfrom
fix/fielddata-empty-key-nil

Conversation

@SharkeyChen

Copy link
Copy Markdown
Collaborator

Previously, buildItem and buildDatasetItem passed empty string as FieldData.Key when constructing dataset items. This caused:

  • Evaluation set items to be silently dropped (due to fd.Key != "" filter)
  • Dataset items to potentially mismatch schema when column names contain CJK characters or spaces

Fix by retrieving the real key from dataset/evaluation-set schema:

  • trace_export_service: use dataset.GetFieldSchemaKeyByName in buildItem
  • auto_evaluate processor: parse schema JSON from TaskRun config to get real keys via fillDatasetKeysFromSchema, avoiding extra RPC calls

What type of PR is this?

Check the PR title

  • This PR title match the format: [<type>][<scope>] <description>. For example: [fix][backend] flaky fix
  • The description of this PR title is user-oriented and clear enough for others to understand.
  • Add documentation if the current PR requires user awareness at the usage level.
  • This PR is written in English. PRs not in English will not be reviewed.

(Optional) Translate the PR title into Chinese

(Optional) More detailed description for this PR(en: English/zh: Chinese)

en:
zh(optional):

(Optional) Which issue(s) this PR fixes

Previously, buildItem and buildDatasetItem passed empty string as FieldData.Key
when constructing dataset items. This caused:
- Evaluation set items to be silently dropped (due to fd.Key != "" filter)
- Dataset items to potentially mismatch schema when column names contain
  CJK characters or spaces

Fix by retrieving the real key from dataset/evaluation-set schema:
- trace_export_service: use dataset.GetFieldSchemaKeyByName in buildItem
- auto_evaluate processor: parse schema JSON from TaskRun config to get
  real keys via fillDatasetKeysFromSchema, avoiding extra RPC calls
Cover fillDatasetKeysFromSchema (empty/invalid JSON, name match, overwrite,
nil key skip), buildItem key propagation, Invoke/BatchInvoke end-to-end key
verification, and trace export service key-from-dataset-schema scenario.
@SharkeyChen SharkeyChen merged commit 8485ee8 into main Jun 24, 2026
10 checks passed
@SharkeyChen SharkeyChen deleted the fix/fielddata-empty-key-nil branch June 24, 2026 12:43
@codecov

codecov Bot commented Jun 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #562      +/-   ##
==========================================
+ Coverage   77.81%   77.82%   +0.01%     
==========================================
  Files         670      670              
  Lines       76100    76151      +51     
==========================================
+ Hits        59217    59268      +51     
+ Misses      13421    13419       -2     
- Partials     3462     3464       +2     
Flag Coverage Δ
unittests 77.82% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...in/task/service/taskexe/processor/auto_evaluate.go 83.43% <100.00%> (+0.13%) ⬆️
...ity/domain/task/service/taskexe/processor/utils.go 100.00% <100.00%> (ø)
...ility/domain/trace/service/trace_export_service.go 74.29% <100.00%> (+0.07%) ⬆️

... and 13 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2011284...907c7e2. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants