Adding makeDiffWorkflow using path to local or alien workflow.config + treeFastCache.C+treeFastCacheND.C

[miranov25]

🛠️ makeDiffWorkflow Utility Function – New Tool for O2 Workflow Comparison

This PR introduces the makeDiffWorkflow shell function, designed to streamline comparison of two workflowconfig.log files — whether local or from Alien.

✅ Function Highlights:

• Compares two JSON workflow configs line-by-line using diff
• Accepts both local and alien:// paths
• Automatically parses logs using makeParse
• Supports filtering by command (default: gpu)
• Supports unified diff or side-by-side diff (default: side-by-side)
• Output is piped to less -R with color for readability

---

🧪 Example Usage

If no arguments are provided:

makeDiffWorkflow

Output:

makeDiffWorkflow: Compare two O2 workflowconfig logs (local or Alien).
Usage:
    makeDiffWorkflow <file0> <file1> <diffType> <filter>
...

---

📄 Full Example – Side-by-Side Diff of GPU Workflows

makeDiffWorkflow alien:///alice/data/2023/LHC23zzk/544515/apass5/1140/o2_ctf_run00544515_orbit0221337280_tf0000047516_epn242/workflowconfig.log alien:///alice/data/2023/LHC23zzk/544515/apass4/1140/o2_ctf_run00544515_orbit0221337280_tf0000047516_epn242/workflowconfig.log 1 o2-gpu  | less

Output:

Fetching file0 from Alien...
Fetching file1 from Alien...
Comparing workflow commands filtered by '^o2-o2-gpu'...
...
{                                                             (
  "command": "o2-gpu-reco-workflow",                          (
  "configKeyValues": {                                        (
    "GPU_global.dEdxDisableResidualGainMap": "1",             (
    "GPU_global.deviceType": "HIP",                           (
ESC[31m    "GPU_proc.conservativeMemoryEstimate": "1",                <
ESC[0m    "GPU_proc.debugLevel": "0",                                 (
    "GPU_proc.deviceNum": "0",                                (
    "GPU_proc.forceHostMemoryPoolSize": "1073741824",         (
    "GPU_proc.forceMemoryPoolSize": "26843545600",            (
    "GPU_proc.ompThreads": "8",                               (
ESC[31m    "GPU_proc.tpcApplyClusterFilterOnCPU": "2",                <
ESC[0m    "GPU_proc.tpcInputWithClusterRejection": "1",       (
ESC[31m    "GPU_proc.tpcUseOldCPUDecoding": "1",                      <
ESC[0m    "GPU_rec_tpc.clusterError2AdditionalYSeeding": "0.1",     (
    "GPU_rec_tpc.clusterError2AdditionalZSeeding": "0.15",    (
    "GPU_rec_tpc.clustersShiftTimebins": "0",                 (
    "GPU_rec_tpc.dEdxClusterRejectionFlagMask": "7",          |     "GPU_rec_tpc.globalTrackingRowRange": "100",
ESC[31m    "GPU_rec_tpc.extrapolationTrackingRowRange": "100",        <
ESC[0mESC[31m    "GPU_rec_tpc.rejectIFCLowRadiusCluster": "1",        <
ESC[0m    "GPU_rec_tpc.trackletMaxSharedFraction": "0.3",             (
    "GPU_rec_tpc.trackletMinSharedNormFactor": "1.",          (
    "TPCCorrMap.lumiInstFactor": "2.414",                     (
    "keyval.input_dir": "/workdir",                           (
    "keyval.output_dir": "/dev/null",                         (
    "trackTuneParams.sourceLevelTPC": "true",                 (
    "trackTuneParams.tpcCovInnerSlope[0]": "1.479465025446798 (
    "trackTuneParams.tpcCovInnerSlope[1]": "5.917860101787194 (
    "trackTuneParams.tpcCovInnerSlope[2]": "8.87679015268079e (
    "trackTuneParams.tpcCovInnerSlope[3]": "3.846609066161676 (
    "trackTuneParams.tpcCovInnerSlope[4]": "1.754653923541247 (
    "trackTuneParams.tpcCovInnerType": "2",                   (
    "trackTuneParams.tpcCovInner[0]": "0.05",                 (
    "trackTuneParams.tpcCovInner[1]": "0.2",                  (
    "trackTuneParams.tpcCovInner[2]": "0.0003",               (
    "trackTuneParams.tpcCovInner[3]": "0.0013",               (
    "trackTuneParams.tpcCovInner[4]": "0.0059300284",         (
    "trackTuneParams.tpcCovOuterSlope[0]": "1.479465025446798 (
    "trackTuneParams.tpcCovOuterSlope[1]": "5.917860101787194 (
    "trackTuneParams.tpcCovOuterSlope[2]": "8.87679015268079e (
    "trackTuneParams.tpcCovOuterSlope[3]": "3.846609066161676 (
    "trackTuneParams.tpcCovOuterSlope[4]": "1.754653923541247 (

---

This tool will save time when debugging config diffs in apass productions or other complex workflows. Let me know if you'd like to adapt it to other use cases like o2-dpl, hlt, etc.

[github-actions]

REQUEST FOR PRODUCTION RELEASES:
To request your PR to be included in production software, please add the corresponding labels called "async-" to your PR. Add the labels directly (if you have the permissions) or add a comment of the form (note that labels are separated by a ",")

+async-label <label1>, <label2>, !<label3> ...

This will add <label1> and <label2> and removes <label3>.

The following labels are available
async-2023-pbpb-apass4
async-2023-pp-apass4
async-2024-pp-apass1
async-2022-pp-apass7
async-2024-pp-cpass0
async-2024-PbPb-apass1
async-2024-ppRef-apass1
async-2024-PbPb-apass2
async-2023-PbPb-apass5

[miranov25]

Hello @shahor02, @chiarazampolli, @swenzel, @matthias-kleiner, @wiechula, and all,

While waiting on the main pull request, I’ve added a utility for fast caching and querying of TTree content — especially useful for cross-correlation, time series, and QA workflows.

This code originated in the private TPC repository, but I’ve now generalized and documented it properly, moving it to O2DPG, with the idea of possibly migrating it to compiled O2 later if needed.

The immediate motivation is to support the QA/QC group with interactive queries on time-dependent data, e.g., retrieving smoothed or reference values via TTree::Draw.

---

🧭 Interface Summary

Function:

int registerMapND(name, tree, exactDims, nearestDim, valueVar, selection);

• name: user-defined string identifier for the map
• tree: source TTree pointer
• exactDims: list of column names for exact-match dimensions (e.g. {"subentry"})
• nearestDim: the coordinate for nearest-neighbor search (typically "time")
• valueVar: variable to interpolate (e.g. "mDCAr_A_NTracks_median")
• selection: TTree selection string (e.g. "1" or "subentry==127")

Alias setup:

setNearestNDAlias(tree, aliasName, name, nearestDimExpr, exactDimExprs);

• aliasName: alias to be created
• name: same map name used in registration
• nearestDimExpr: expression (e.g. "time")
• exactDimExprs: list of expressions for exact-match dimensions

---

🧪 Example: ND Lookup (e.g. HV scan QA)

void exampleND() {
  TTree* tree = new TTree("tree", "demo");
  int mapID = registerMapND("test_map", tree, {"CRU", "iTF"}, "time", "val", "subentry==127");
  setNearestNDAlias(tree, "val_interp", "test_map", "time", {"CRU", "iTF"});
  tree->Draw("val:val_interp", "val!=0", "colz");
}

🕒 Example: Time-Series QA Overlay

void exampleTimeSeries() {
  TFile *f = TFile::Open("timeSeries10000_apass5.root");
  TTree *tree = (TTree*)f->Get("timeSeries");
  int mapID = registerMapND("dcar_vs_time", tree, {"subentry"}, "time", "mTSITSTPC.mDCAr_A_NTracks_median", "1");
  setNearestNDAlias(tree, "mDCAr_A_NTracks_median_interp", "dcar_vs_time", "time", {"subentry"});
  tree->Draw("mTSITSTPC.mDCAr_A_NTracks_median:mDCAr_A_NTracks_median_interp", "indexType==1", "", 10000);
}

---

📄 For a more detailed explanation and API overview, see:
[README_treeFastCache.md](https://github.com/miranov25/O2DPG/blob/master/UTILS/Parsers/README_treeFastCache.md)

Let me know if you'd like more examples or integration into compiled O2 code.

Best regards,
Marian

[miranov25]

Hallo @sawenzel @pzhristov @davidrohr @shahor02, @chiarazampolli

Could one of you please approve this MR?
I’d like to move ahead with integrating the treeCache macros into O2DPG and continue their development and usage. Once the interface is stable we’ll port the implementation to compiled O2 code, which feels like the right long-term home.

Thanks!