Skip to content

Backport work around #3744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 135 commits into from

Conversation

xinyual
Copy link
Contributor

@xinyual xinyual commented Jun 6, 2025

Description

[Describe what this change achieves]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

currantw and others added 30 commits June 1, 2025 13:50
…ject#3075)

Signed-off-by: Norman Jordan <[email protected]>
Signed-off-by: normanj-bitquill <[email protected]>
Co-authored-by: Andrew Carbonetto <[email protected]>
* Add trendline (With SWA) PPL command

---------

Signed-off-by: James Duong <[email protected]>
Signed-off-by: Andrew Carbonetto <[email protected]>
Co-authored-by: Andrew Carbonetto <[email protected]>
Signed-off-by: currantw <[email protected]>

* Add support for casting (`cast(field_name to ip)`) and remove existing unused sorting syntax.

Signed-off-by: currantw <[email protected]>

* Update comparison logic to compare in IPv6

Signed-off-by: currantw <[email protected]>

* Fix bug casting to IP

Signed-off-by: currantw <[email protected]>

* Fix failing tests

Signed-off-by: currantw <[email protected]>

* Assert that comparison only valid if same type, update tests accordingly

Signed-off-by: currantw <[email protected]>

* Add additional tests to increase code coverage

Signed-off-by: currantw <[email protected]>

* Integrate `cidrmatch` changes

Signed-off-by: currantw <[email protected]>

* Remove `OpenSearchIPType` data type

Signed-off-by: currantw <[email protected]>

* Fix more failing tests

Signed-off-by: currantw <[email protected]>

* Minor cleanup

Signed-off-by: currantw <[email protected]>

* Add new tests for IP data type to `SortCommandIT`, and update `weblogs` test data.

Signed-off-by: currantw <[email protected]>

* Fixing IT test failure.

Signed-off-by: currantw <[email protected]>

* Spotless and update test to sort in SQL

Signed-off-by: currantw <[email protected]>

* Fix broken link

Signed-off-by: currantw <[email protected]>

* Fix failing code coverage

Signed-off-by: currantw <[email protected]>

* Fix failing doctest

Signed-off-by: currantw <[email protected]>

* Fix failing `ip.rst` doctest

Signed-off-by: currantw <[email protected]>

* Fix test failure due to merge.

Signed-off-by: currantw <[email protected]>

* Fix spotless

Signed-off-by: currantw <[email protected]>

* Add missing `url` field

Signed-off-by: currantw <[email protected]>

* Address minor review comments.

Signed-off-by: currantw <[email protected]>

* Revert sort syntax changes

Signed-off-by: currantw <[email protected]>

* Minor doc update

Signed-off-by: currantw <[email protected]>

* FIx failing `ip.rst` doctest

Signed-off-by: currantw <[email protected]>

* Add `IPComparisonIT` tests for comparison operators, rename modules and weblogs test index to make plural for consistency.

Signed-off-by: currantw <[email protected]>

---------

Signed-off-by: currantw <[email protected]>
…er (opensearch-project#3249)

* First commit for Calcite integration

Signed-off-by: Lantao Jin <[email protected]>

* disable java security manager in IT

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
…ject#3258)

* [POC] Make Calcite execute successfully

Signed-off-by: Heng Qian <[email protected]>

* [POC] Change caching schema to simple schema and avoid registering table when visitRelation.

Signed-off-by: Heng Qian <[email protected]>

* spotlessApply

Signed-off-by: Heng Qian <[email protected]>

* address comments

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
* Make basic aggregation working (partial)

Signed-off-by: Lantao Jin <[email protected]>

* add a settings to enable calcite

Signed-off-by: Lantao Jin <[email protected]>

* add more UTs

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
opensearch-project#3327)

* Support Filter and Project pushdown

Signed-off-by: Heng Qian <[email protected]>

* Support Filter and Project pushdown v2

Signed-off-by: Heng Qian <[email protected]>

* Address comments

Signed-off-by: Heng Qian <[email protected]>

* Add original license for PredicateAnalyzer

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
* Build integration test framework

Signed-off-by: Lantao Jin <[email protected]>

* make local work

Signed-off-by: Lantao Jin <[email protected]>

* Fix the timestamp issue

Signed-off-by: Lantao Jin <[email protected]>

* address comments

Signed-off-by: Lantao Jin <[email protected]>

* fix java style and rename CalcitePPLTestCase back to CalcitePPLIntegTestCase

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
…oject#3355)

* Add more aggregation tests

Signed-off-by: Lantao Jin <[email protected]>

* delete unrelavant code

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
* Transform to calcite plan before executing

Signed-off-by: Heng Qian <[email protected]>

* Fix bug for single column row

Signed-off-by: Heng Qian <[email protected]>

* Add settings for calcite pushdown

Signed-off-by: Heng Qian <[email protected]>

* Lazily construct OpenSearchRequestBuilder and do push down

Signed-off-by: Heng Qian <[email protected]>

* Address comments and disable push down

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
* Fix PredicateAnalyzer for in and notIn

Signed-off-by: Heng Qian <[email protected]>

* Change text field to keyword since we don't support push down for that type

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
…3376)

* [BugFix] Fix text field push down

Signed-off-by: Heng Qian <[email protected]>

* Ignore CalciteSortCommandIT.testSortWithNullValue

Signed-off-by: Heng Qian <[email protected]>

* Refine code: only get keyword subfield for termQuery builder

Signed-off-by: Heng Qian <[email protected]>

* Refine code

Signed-off-by: Heng Qian <[email protected]>

* remove ignore tests in CalcitePPLInSubqueryIT

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
* add udf/udaf interface and take/sqrt function

Signed-off-by: xinyual <[email protected]>

* add UT

Signed-off-by: xinyual <[email protected]>

* add POW, Atan, Atan2 and corresponding UT

Signed-off-by: xinyual <[email protected]>

* apply spotless

Signed-off-by: xinyual <[email protected]>

* fix table for join it

Signed-off-by: xinyual <[email protected]>

* add java doc

Signed-off-by: xinyual <[email protected]>

* apply spotless

Signed-off-by: xinyual <[email protected]>

---------

Signed-off-by: xinyual <[email protected]>
…t#3392)

* Implement ppl scalar subquery command with Calcite

Signed-off-by: Lantao Jin <[email protected]>

* more general subquery checker

Signed-off-by: Lantao Jin <[email protected]>

* support correlated IN subquery

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
* Change push down to logical index scan

Signed-off-by: Heng Qian <[email protected]>

* Support Aggregate Push Down

Signed-off-by: Heng Qian <[email protected]>

* Rebase and resolve conflict

Signed-off-by: Heng Qian <[email protected]>

* Add TODO

Signed-off-by: Heng Qian <[email protected]>

* Address comments

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
* add string udfs

Signed-off-by: xinyual <[email protected]>

* add it to string

Signed-off-by: xinyual <[email protected]>

* add IT for string function

Signed-off-by: xinyual <[email protected]>

* remove change for local test

Signed-off-by: xinyual <[email protected]>

* revert change

Signed-off-by: xinyual <[email protected]>

---------

Signed-off-by: xinyual <[email protected]>
…nsearch-project#3405)

* Keep aggregation in Calcite consistent with current PPL behavior

Signed-off-by: Lantao Jin <[email protected]>

* remove unrelated code

Signed-off-by: Lantao Jin <[email protected]>

* revert some code

Signed-off-by: Lantao Jin <[email protected]>

* fix issue 3404

Signed-off-by: Lantao Jin <[email protected]>

* add more tests

Signed-off-by: Lantao Jin <[email protected]>

* address comments

Signed-off-by: Lantao Jin <[email protected]>

* add more tests

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
* Support multiple table and index pattern

Signed-off-by: Heng Qian <[email protected]>

* Fix UT

Signed-off-by: Heng Qian <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
* add condition udfs

Signed-off-by: xinyual <[email protected]>

* add IT for conditions and register null table

Signed-off-by: xinyual <[email protected]>

* fix it

Signed-off-by: xinyual <[email protected]>

* update utils define

Signed-off-by: xinyual <[email protected]>

* add condition functions

Signed-off-by: xinyual <[email protected]>

* modify IT

Signed-off-by: xinyual <[email protected]>

* fix IT

Signed-off-by: xinyual <[email protected]>

* revert useless change and add comments

Signed-off-by: xinyual <[email protected]>

* reverse typo and apply spotless

Signed-off-by: xinyual <[email protected]>

---------

Signed-off-by: xinyual <[email protected]>
xinyual and others added 29 commits June 3, 2025 17:26
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
* Support Grok command in Calcite engine

Signed-off-by: Songkan Tang <[email protected]>

* Register SqlStdOperatorTable.ITEM into PPLFuncImpTable

Signed-off-by: Songkan Tang <[email protected]>

* Fix case function ITs caused by weblogs index data change

Signed-off-by: Songkan Tang <[email protected]>

* Enforce Locale attempt to fix flaky grok compilation

Signed-off-by: Songkan Tang <[email protected]>

* Fix checkstyle

Signed-off-by: Songkan Tang <[email protected]>

* Add anonymizer test for grok command

Signed-off-by: Songkan Tang <[email protected]>

---------

Signed-off-by: Songkan Tang <[email protected]>
* Support ResourceMonitor with Calcite

Signed-off-by: Lantao Jin <[email protected]>

* fix IT

Signed-off-by: Lantao Jin <[email protected]>

* fix UT related to decimal literal

Signed-off-by: Lantao Jin <[email protected]>

---------

Signed-off-by: Lantao Jin <[email protected]>
…ch-project#3626)

* Support type checker For PPL functions

Signed-off-by: Heng Qian <[email protected]>

* Implement type checkers for UDF

Signed-off-by: Yuanchun Shen <[email protected]>

* Implement composite type checker

Signed-off-by: Yuanchun Shen <[email protected]>

* Define type checkers for UDFs

Signed-off-by: Yuanchun Shen <[email protected]>

* Display expected signatures when type checking fails & support type checking for non-operator registrated udf

Signed-off-by: Yuanchun Shen <[email protected]>

* Add ITs for UDF type checking

Signed-off-by: Yuanchun Shen <[email protected]>

* Fix: correct CONV's operand types

Signed-off-by: Yuanchun Shen <[email protected]>

* Fix test parse IT

Signed-off-by: Yuanchun Shen <[email protected]>

* Remove unused code in UserDefinedFunctionUtils

Signed-off-by: Yuanchun Shen <[email protected]>

* Refactor: improve ITs, javadocs, type checkers

Signed-off-by: Yuanchun Shen <[email protected]>

* Improve javadoc for PPLTypeChecker

Signed-off-by: Yuanchun Shen <[email protected]>

* Return false at validation for mismatched operand count with expectations
- additionally fixed span type checker

Signed-off-by: Yuanchun Shen <[email protected]>

* Add type checker for cidrmatch

Signed-off-by: Yuanchun Shen <[email protected]>

* Correct cidrmatch type checker

Signed-off-by: Yuanchun Shen <[email protected]>

* Fix compile error in CalcitePPLAbstractTest.java

Signed-off-by: Yuanchun Shen <[email protected]>

* Experiment: support type checking for built-in operators by accessing composition field of CompositeOperandTypeChecker via relection. May cause fatal errors if IllegalAccessException is thrown.

Signed-off-by: Yuanchun Shen <[email protected]>

* Check composition type for only calcite's built-in operators

Signed-off-by: Yuanchun Shen <[email protected]>

* Support parameter validation for comparabale operators

Signed-off-by: Yuanchun Shen <[email protected]>

* Support parameter validation for coalesce (by reusing comparable type checker)

Signed-off-by: Yuanchun Shen <[email protected]>

* Support parameter validation for substring

Signed-off-by: Yuanchun Shen <[email protected]>

* Support parameter validation for if

Signed-off-by: Yuanchun Shen <[email protected]>

* Support parameter validation for nullif, isempty, isblank

Signed-off-by: Yuanchun Shen <[email protected]>

* Align types in type check error information to actual types in PPL

Signed-off-by: Yuanchun Shen <[email protected]>

* Display PPL types when failing to resolve a function

Signed-off-by: Yuanchun Shen <[email protected]>

* Create type checker for geoip function

Signed-off-by: Yuanchun Shen <[email protected]>

* Define type checker for grok and item operators

Signed-off-by: Yuanchun Shen <[email protected]>

---------

Signed-off-by: Heng Qian <[email protected]>
Signed-off-by: Yuanchun Shen <[email protected]>
Co-authored-by: Heng Qian <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
Signed-off-by: xinyual <[email protected]>
@xinyual xinyual closed this Jun 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.