feat: Add MERGE (UPSERT) support to Oracle Sink#636
Draft
cjac wants to merge 1 commit intodata-integrations:developfrom
Draft
feat: Add MERGE (UPSERT) support to Oracle Sink#636cjac wants to merge 1 commit intodata-integrations:developfrom
cjac wants to merge 1 commit intodata-integrations:developfrom
Conversation
This commit enhances the Oracle Sink plugin to support `MERGE` operations, providing an idempotent way to handle data writes, especially in scenarios involving Spark task retries. New Features: - **UPSERT Operation:** A new `operation` property in the sink configuration allows users to specify "upsert". - **Merge Keys:** A `mergeKeys` property is introduced to define the column(s) for the `MERGE` statement's `ON` clause. - **`OracleETLDBOutputFormat`:** A new class extending `ETLDBOutputFormat` to construct the Oracle-specific `MERGE` SQL statement using a `USING DUAL` approach. - **Conditional Logic:** The sink now conditionally uses `OracleETLDBOutputFormat` when the "upsert" operation is selected. Test Improvements: - Updated `docker-compose.yml` to use `container-registry.oracle.com/database/free:latest` with healthcheck. - README.md updated to reflect new Oracle image and Podman instructions. - Increased test timeout in `DatabasePluginTestBase`. - Added `testDBSinkUniqueConstraintViolation` to `OracleSinkTestRun` to demonstrate ORA-00001 on duplicate INSERTs. - Added `testDBSinkMerge` to `OracleSinkTestRun` to validate the new UPSERT functionality. - Commented out `apache-rat-plugin` in the root `pom.xml` to potentially speed up local builds. This change addresses issues where task retries could lead to unique constraint violations (ORA-00001) by providing a true upsert mechanism.
Author
|
Internal notes: go/sf/64273740 |
| MySQL, Postgresql, MSSQL, DB2, MemSQL, SAP HANA, and Oracle are using prebuilt images. | ||
|
|
||
| Oracle DB image should be build separately. | ||
| MemSQL image should be configure after start. |
Contributor
There was a problem hiding this comment.
Why these are added?
| * License for the specific language governing permissions and limitations under | ||
| * the License. | ||
| */ | ||
|
|
Contributor
There was a problem hiding this comment.
No need to remove this
| } | ||
|
|
||
| @Override | ||
| protected void insertOperation(PreparedStatement stmt) throws SQLException { |
Contributor
There was a problem hiding this comment.
why this method is removed?
| @@ -0,0 +1,63 @@ | |||
| import java.sql.Connection; | |||
Contributor
There was a problem hiding this comment.
how is this relevant to plugin code
| super.test(JDBC_DRIVER_CLASS_NAME, connector, "Failed to create connection to database via connection string:" + | ||
| " jdbc:oracle:thin:@localhost:1521:database and arguments: " + | ||
| "{user=username, oracle.jdbc.timezoneAsRegion=false, " + | ||
| "{oracle.jdbc.timezoneAsRegion=false, user=username, " + |
Contributor
There was a problem hiding this comment.
why sequence is reversed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit enhances the Oracle Sink plugin to support
MERGEoperations, providing an idempotent way to handle data writes, especially in scenarios involving Spark task retries.New Features:
operationproperty in the sink configuration allows users to specify "upsert".mergeKeysproperty is introduced to define the column(s) for theMERGEstatement'sONclause.OracleETLDBOutputFormat: A new class extendingETLDBOutputFormatto construct the Oracle-specificMERGESQL statement using aUSING DUALapproach.OracleETLDBOutputFormatwhen the "upsert" operation is selected.Test Improvements:
docker-compose.ymlto usecontainer-registry.oracle.com/database/free:latestwith healthcheck.DatabasePluginTestBase.testDBSinkUniqueConstraintViolationtoOracleSinkTestRunto demonstrate ORA-00001 on duplicate INSERTs.testDBSinkMergetoOracleSinkTestRunto validate the new UPSERT functionality.apache-rat-pluginin the rootpom.xmlto potentially speed up local builds.This change addresses issues where task retries could lead to unique constraint violations (ORA-00001) by providing a true upsert mechanism.