Skip to content

[BugFix] Prevent push down limit with offset reach maxResultWindow #3713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 10, 2025

Conversation

qianheng-aws
Copy link
Collaborator

Description

Prevent push down limit with offset reach maxResultWindow

The index setting of max_result_window is the maximum restriction of from + size.
https://docs.opensearch.org/docs/latest/install-and-configure/configuring-opensearch/index-settings/

Thus in our code of building OpenSearch request, we have logic of using size equals to maxResultWindow -statFrom if startFrom + size > maxResultWindow. It will have problem if we got new size equal to 0 or less than 0.

We could address this issue by preventing such limit from push down if it would cause the new startFrom to reach the maxResultWindow.

Related Issues

Resolves #3102

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Comment on lines 25 to 31
teardown:
- do:
query.settings:
body:
transient:
plugins.calcite.enabled : false
plugins.calcite.fallback.allowed : true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why we need this teardown step? it seems not enable calcite in setup step.

@penghuo is each xyx.yml independent?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need revert setting, by default is disabled

query: 'source=test | head 2 | head 1 from 1 '
- match: { "total": 1 }
- match: { "schema": [ { "name": "id", "type": "bigint" } ] }
- match: { "datarows": [ [ 2 ] ] }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the result without this fixing?

Comment on lines 25 to 31
teardown:
- do:
query.settings:
body:
transient:
plugins.calcite.enabled : false
plugins.calcite.fallback.allowed : true
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, no need revert setting, by default is disabled

Comment on lines 25 to 31
teardown:
- do:
query.settings:
body:
transient:
plugins.calcite.enabled : false
plugins.calcite.fallback.allowed : true
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need revert setting, by default is disabled

Signed-off-by: Heng Qian <[email protected]>
@penghuo penghuo merged commit 066e429 into opensearch-project:main Jun 10, 2025
22 checks passed
penghuo pushed a commit that referenced this pull request Jun 16, 2025
…3713)

* [BugFix] Prevent push down limit with offset no less than maxResultWindow

Signed-off-by: Heng Qian <[email protected]>

* Address comments

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: xinyual <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] PPL head limit and SQL limit offset throw exception
3 participants