Fix IncrementAnIndexPaginationIterator#61
Fix IncrementAnIndexPaginationIterator#61VictorD-Veolia wants to merge 1 commit intodata-integrations:developfrom
Conversation
|
@VictorD-Veolia: possible to share the exact problem that this fixes? Some sample response that this fails for? Also, could you add a test? |
|
The "Increment An Index" pagination is not usable at all. Having a pipeline with an HTTP plugin using "Increment an Index" pagination will throw "Failed to configure pipeline: Stage 'HTTP' encountered : Null error occurred while configuring the stage HTTP." at pipeline startup (with every configuration and independently of input data). |
rmstar
left a comment
rmstar
left a comment
There was a problem hiding this comment.
- Please create a Jira that describes the issue and link to it on this PR
- Please add a unit test
| import java.util.Map; | ||
|
|
||
| /** | ||
| * Returns elements from json one by one by given json path. |
| if (json.isJsonObject()) { | ||
| JSONUtil.JsonQueryResponse queryResponse = | ||
| JSONUtil.getJsonElementByPath(json.getAsJsonObject(), config.getResultPath(), optionalFields); | ||
| JSONUtil.getJsonElementByPath(json.getAsJsonObject(), config.getResultPath(), optionalFields); |
| } else { | ||
| throw new IllegalArgumentException(String.format("Element found by '%s' json path is expected to be an object " + | ||
| "or an array. Primitive found", config.getResultPath())); | ||
| "or an array. Primitive found", config.getResultPath())); |
| * | ||
| * <p> | ||
| * Example next element: | ||
| * { |
| * | ||
| * <p> | ||
| * { | ||
| * "key":"NETTY-13", |
| if (json.isJsonObject()) { | ||
| JSONUtil.JsonQueryResponse queryResponse = JSONUtil.getJsonElementByPath(json.getAsJsonObject(), | ||
| path, optionalFields); | ||
| path, optionalFields); |
| JsonPage(BaseHttpSourceConfig config, HttpResponse httpResponse) { | ||
| super(httpResponse); | ||
| this.config = config; | ||
| String body = httpResponse.getBody(); |
| private final BaseHttpSourceConfig config; | ||
| private final List<String> optionalFields; | ||
|
|
||
| // TODO : handle case where the result json object is null |
There was a problem hiding this comment.
what happens when result json object is null? are you planning to handle that case in another PR?
| InvalidEntry<StructuredRecord> error = | ||
| new InvalidEntry<>(1, "Couldn't find all required fields in the record", record); | ||
| return new PageEntry(error, config.getErrorHandling()); | ||
| if (iterator.hasNext()) { |
There was a problem hiding this comment.
you can avoid one level of indent if you return when there are no entries:
if (!iterartor.hasNext()) {
return new PageEntry(null);
}
...
3 participants
Fix "Increment An Index" Pagination to match the expected behavior when "maxIndex" is null ("If empty, pagination will happen until the page with no elements.").