feat(amber): Re-enable operator reconfiguration in Amber

[shengquan-ni]

What changes were proposed in this PR?

Per the discussion in #4016, we decided to bring the operator reconfiguration feature back to the Amber engine. This PR includes only the backend changes for this feature, but it is enabled on both the Java and Python sides.

Since the code for the Fries Algorithm is still in the codebase, this feature is relatively straightforward to implement and maintain going forward.

This PR allows source operators to be included in the reconfiguration scope (MCS), but it does not allow source operators themselves to be modified. First, under the current iterator-based interface, the state of a source operator is fully encapsulated within its iterator. Reading or manipulating the iterator state is already very difficult in both Scala and Python. Second, even if we could access the state, it would still be hard for users to clearly define the expected state transition semantics—e.g., whether to preserve the old state, reset it, or partially transfer it to the new operator.

Due to the reasons above, we disable reconfiguration of source operators for now. If clear use cases emerge in the future, we can revisit this design decision.

Any related issues, documentation, discussions?

See #4016.

How was this PR tested?

Introduced unit tests for this feature.

This PR also updates scala CI to install python dependencies as we are using Python UDFs in our e2e tests.

Was this PR authored or co-authored using generative AI tooling?

No

[chenlica]

Thanks, @shengquan-ni. Can we get rid of the restriction of "not including source operators"? This restriction could suggest some part of our design is not elegant. We can discuss the complexity of removing this restriction. If needed, we can schedule a call to discuss.

I am adding @zuozhiw so that he can review this PR as well.

[shengquan-ni]

Thanks, @shengquan-ni. Can we get rid of the restriction of "not including source operators"? This restriction could suggest some part of our design is not elegant. We can discuss the complexity of removing this restriction. If needed, we can schedule a call to discuss.

I am adding @zuozhiw so that he can review this PR as well.

Earlier, I mentioned that the MCS could not include source operators due to certain ECM constraints. However, I have since identified a simple solution to address that issue, so this restriction has now been removed.

That said, reconfiguring source operators themselves is still not supported. I’ve updated the PR description to explain the reasoning behind this decision.

We can further discuss this if needed.

[chenlica]

Thank you, @shengquan-ni .

@Yicong-Huang @zuozhiw and @aglinxinyuan : can you chime in?