Backport work around #3744

xinyual · 2025-06-06T06:12:21Z

Description

[Describe what this change achieves]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…t#3110) Signed-off-by: currantw <[email protected]>

…ject#3075) Signed-off-by: Norman Jordan <[email protected]> Signed-off-by: normanj-bitquill <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]>

* Add trendline (With SWA) PPL command --------- Signed-off-by: James Duong <[email protected]> Signed-off-by: Andrew Carbonetto <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]>

Signed-off-by: currantw <[email protected]> * Add support for casting (`cast(field_name to ip)`) and remove existing unused sorting syntax. Signed-off-by: currantw <[email protected]> * Update comparison logic to compare in IPv6 Signed-off-by: currantw <[email protected]> * Fix bug casting to IP Signed-off-by: currantw <[email protected]> * Fix failing tests Signed-off-by: currantw <[email protected]> * Assert that comparison only valid if same type, update tests accordingly Signed-off-by: currantw <[email protected]> * Add additional tests to increase code coverage Signed-off-by: currantw <[email protected]> * Integrate `cidrmatch` changes Signed-off-by: currantw <[email protected]> * Remove `OpenSearchIPType` data type Signed-off-by: currantw <[email protected]> * Fix more failing tests Signed-off-by: currantw <[email protected]> * Minor cleanup Signed-off-by: currantw <[email protected]> * Add new tests for IP data type to `SortCommandIT`, and update `weblogs` test data. Signed-off-by: currantw <[email protected]> * Fixing IT test failure. Signed-off-by: currantw <[email protected]> * Spotless and update test to sort in SQL Signed-off-by: currantw <[email protected]> * Fix broken link Signed-off-by: currantw <[email protected]> * Fix failing code coverage Signed-off-by: currantw <[email protected]> * Fix failing doctest Signed-off-by: currantw <[email protected]> * Fix failing `ip.rst` doctest Signed-off-by: currantw <[email protected]> * Fix test failure due to merge. Signed-off-by: currantw <[email protected]> * Fix spotless Signed-off-by: currantw <[email protected]> * Add missing `url` field Signed-off-by: currantw <[email protected]> * Address minor review comments. Signed-off-by: currantw <[email protected]> * Revert sort syntax changes Signed-off-by: currantw <[email protected]> * Minor doc update Signed-off-by: currantw <[email protected]> * FIx failing `ip.rst` doctest Signed-off-by: currantw <[email protected]> * Add `IPComparisonIT` tests for comparison operators, rename modules and weblogs test index to make plural for consistency. Signed-off-by: currantw <[email protected]> --------- Signed-off-by: currantw <[email protected]>

…er (opensearch-project#3249) * First commit for Calcite integration Signed-off-by: Lantao Jin <[email protected]> * disable java security manager in IT Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]>

…ject#3258) * [POC] Make Calcite execute successfully Signed-off-by: Heng Qian <[email protected]> * [POC] Change caching schema to simple schema and avoid registering table when visitRelation. Signed-off-by: Heng Qian <[email protected]> * spotlessApply Signed-off-by: Heng Qian <[email protected]> * address comments Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]>

* Make basic aggregation working (partial) Signed-off-by: Lantao Jin <[email protected]> * add a settings to enable calcite Signed-off-by: Lantao Jin <[email protected]> * add more UTs Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]>

opensearch-project#3327) * Support Filter and Project pushdown Signed-off-by: Heng Qian <[email protected]> * Support Filter and Project pushdown v2 Signed-off-by: Heng Qian <[email protected]> * Address comments Signed-off-by: Heng Qian <[email protected]> * Add original license for PredicateAnalyzer Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]>

* Build integration test framework Signed-off-by: Lantao Jin <[email protected]> * make local work Signed-off-by: Lantao Jin <[email protected]> * Fix the timestamp issue Signed-off-by: Lantao Jin <[email protected]> * address comments Signed-off-by: Lantao Jin <[email protected]> * fix java style and rename CalcitePPLTestCase back to CalcitePPLIntegTestCase Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]>

…nsearch-project#3349) Signed-off-by: Lantao Jin <[email protected]>

…oject#3355) * Add more aggregation tests Signed-off-by: Lantao Jin <[email protected]> * delete unrelavant code Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]>

* Transform to calcite plan before executing Signed-off-by: Heng Qian <[email protected]> * Fix bug for single column row Signed-off-by: Heng Qian <[email protected]> * Add settings for calcite pushdown Signed-off-by: Heng Qian <[email protected]> * Lazily construct OpenSearchRequestBuilder and do push down Signed-off-by: Heng Qian <[email protected]> * Address comments and disable push down Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]>

Signed-off-by: Lantao Jin <[email protected]>

Signed-off-by: Heng Qian <[email protected]>

* Fix PredicateAnalyzer for in and notIn Signed-off-by: Heng Qian <[email protected]> * Change text field to keyword since we don't support push down for that type Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]>

…3371)

Signed-off-by: xinyual <[email protected]>

…3376) * [BugFix] Fix text field push down Signed-off-by: Heng Qian <[email protected]> * Ignore CalciteSortCommandIT.testSortWithNullValue Signed-off-by: Heng Qian <[email protected]> * Refine code: only get keyword subfield for termQuery builder Signed-off-by: Heng Qian <[email protected]> * Refine code Signed-off-by: Heng Qian <[email protected]> * remove ignore tests in CalcitePPLInSubqueryIT Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]>

* add udf/udaf interface and take/sqrt function Signed-off-by: xinyual <[email protected]> * add UT Signed-off-by: xinyual <[email protected]> * add POW, Atan, Atan2 and corresponding UT Signed-off-by: xinyual <[email protected]> * apply spotless Signed-off-by: xinyual <[email protected]> * fix table for join it Signed-off-by: xinyual <[email protected]> * add java doc Signed-off-by: xinyual <[email protected]> * apply spotless Signed-off-by: xinyual <[email protected]> --------- Signed-off-by: xinyual <[email protected]>

…ry impl (opensearch-project#3377)

…ect#3378) Signed-off-by: Lantao Jin <[email protected]>

…t#3388)

…t#3392) * Implement ppl scalar subquery command with Calcite Signed-off-by: Lantao Jin <[email protected]> * more general subquery checker Signed-off-by: Lantao Jin <[email protected]> * support correlated IN subquery Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]>

* Change push down to logical index scan Signed-off-by: Heng Qian <[email protected]> * Support Aggregate Push Down Signed-off-by: Heng Qian <[email protected]> * Rebase and resolve conflict Signed-off-by: Heng Qian <[email protected]> * Add TODO Signed-off-by: Heng Qian <[email protected]> * Address comments Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]>

* add string udfs Signed-off-by: xinyual <[email protected]> * add it to string Signed-off-by: xinyual <[email protected]> * add IT for string function Signed-off-by: xinyual <[email protected]> * remove change for local test Signed-off-by: xinyual <[email protected]> * revert change Signed-off-by: xinyual <[email protected]> --------- Signed-off-by: xinyual <[email protected]>

…nsearch-project#3405) * Keep aggregation in Calcite consistent with current PPL behavior Signed-off-by: Lantao Jin <[email protected]> * remove unrelated code Signed-off-by: Lantao Jin <[email protected]> * revert some code Signed-off-by: Lantao Jin <[email protected]> * fix issue 3404 Signed-off-by: Lantao Jin <[email protected]> * add more tests Signed-off-by: Lantao Jin <[email protected]> * address comments Signed-off-by: Lantao Jin <[email protected]> * add more tests Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]>

* Support multiple table and index pattern Signed-off-by: Heng Qian <[email protected]> * Fix UT Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]>

Signed-off-by: Lantao Jin <[email protected]>

* add condition udfs Signed-off-by: xinyual <[email protected]> * add IT for conditions and register null table Signed-off-by: xinyual <[email protected]> * fix it Signed-off-by: xinyual <[email protected]> * update utils define Signed-off-by: xinyual <[email protected]> * add condition functions Signed-off-by: xinyual <[email protected]> * modify IT Signed-off-by: xinyual <[email protected]> * fix IT Signed-off-by: xinyual <[email protected]> * revert useless change and add comments Signed-off-by: xinyual <[email protected]> * reverse typo and apply spotless Signed-off-by: xinyual <[email protected]> --------- Signed-off-by: xinyual <[email protected]>

* Support Limit pushdown Signed-off-by: Yuanchun Shen <[email protected]> * Minor: add copyright header Signed-off-by: Yuanchun Shen <[email protected]> * Refactor: create a isLogicalSortLimit method to enhance readability Signed-off-by: Yuanchun Shen <[email protected]> * Disable limit-then-filter pushdown with Calcite Signed-off-by: Yuanchun Shen <[email protected]> * Disable limit-then-agg pushdown & add ITs for multiple head clauses Signed-off-by: Yuanchun Shen <[email protected]> * Add limit-filter-limit pushdown IT Signed-off-by: Yuanchun Shen <[email protected]> * Disable limit-then-filter pushdown without Calcite Signed-off-by: Yuanchun Shen <[email protected]> * Move predicates of limit pushdown ahead Signed-off-by: Yuanchun Shen <[email protected]> * Accumulate offsets when pushing down limit Signed-off-by: Yuanchun Shen <[email protected]> * Improve error message when failed to pushdown limit Signed-off-by: Yuanchun Shen <[email protected]> * Update limit pushdown error message Signed-off-by: Yuanchun Shen <[email protected]> * Fix multiple limit pushdown with shirinking caused by offset Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]>

Signed-off-by: xinyual <[email protected]>

…th OpenSearch (opensearch-project#3698)" This reverts commit 0defdcd.

Signed-off-by: xinyual <[email protected]>

--------- Signed-off-by: Heng Qian <[email protected]>

Signed-off-by: xinyual <[email protected]>

* Support Grok command in Calcite engine Signed-off-by: Songkan Tang <[email protected]> * Register SqlStdOperatorTable.ITEM into PPLFuncImpTable Signed-off-by: Songkan Tang <[email protected]> * Fix case function ITs caused by weblogs index data change Signed-off-by: Songkan Tang <[email protected]> * Enforce Locale attempt to fix flaky grok compilation Signed-off-by: Songkan Tang <[email protected]> * Fix checkstyle Signed-off-by: Songkan Tang <[email protected]> * Add anonymizer test for grok command Signed-off-by: Songkan Tang <[email protected]> --------- Signed-off-by: Songkan Tang <[email protected]>

* Support ResourceMonitor with Calcite Signed-off-by: Lantao Jin <[email protected]> * fix IT Signed-off-by: Lantao Jin <[email protected]> * fix UT related to decimal literal Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]>

…ch-project#3626) * Support type checker For PPL functions Signed-off-by: Heng Qian <[email protected]> * Implement type checkers for UDF Signed-off-by: Yuanchun Shen <[email protected]> * Implement composite type checker Signed-off-by: Yuanchun Shen <[email protected]> * Define type checkers for UDFs Signed-off-by: Yuanchun Shen <[email protected]> * Display expected signatures when type checking fails & support type checking for non-operator registrated udf Signed-off-by: Yuanchun Shen <[email protected]> * Add ITs for UDF type checking Signed-off-by: Yuanchun Shen <[email protected]> * Fix: correct CONV's operand types Signed-off-by: Yuanchun Shen <[email protected]> * Fix test parse IT Signed-off-by: Yuanchun Shen <[email protected]> * Remove unused code in UserDefinedFunctionUtils Signed-off-by: Yuanchun Shen <[email protected]> * Refactor: improve ITs, javadocs, type checkers Signed-off-by: Yuanchun Shen <[email protected]> * Improve javadoc for PPLTypeChecker Signed-off-by: Yuanchun Shen <[email protected]> * Return false at validation for mismatched operand count with expectations - additionally fixed span type checker Signed-off-by: Yuanchun Shen <[email protected]> * Add type checker for cidrmatch Signed-off-by: Yuanchun Shen <[email protected]> * Correct cidrmatch type checker Signed-off-by: Yuanchun Shen <[email protected]> * Fix compile error in CalcitePPLAbstractTest.java Signed-off-by: Yuanchun Shen <[email protected]> * Experiment: support type checking for built-in operators by accessing composition field of CompositeOperandTypeChecker via relection. May cause fatal errors if IllegalAccessException is thrown. Signed-off-by: Yuanchun Shen <[email protected]> * Check composition type for only calcite's built-in operators Signed-off-by: Yuanchun Shen <[email protected]> * Support parameter validation for comparabale operators Signed-off-by: Yuanchun Shen <[email protected]> * Support parameter validation for coalesce (by reusing comparable type checker) Signed-off-by: Yuanchun Shen <[email protected]> * Support parameter validation for substring Signed-off-by: Yuanchun Shen <[email protected]> * Support parameter validation for if Signed-off-by: Yuanchun Shen <[email protected]> * Support parameter validation for nullif, isempty, isblank Signed-off-by: Yuanchun Shen <[email protected]> * Align types in type check error information to actual types in PPL Signed-off-by: Yuanchun Shen <[email protected]> * Display PPL types when failing to resolve a function Signed-off-by: Yuanchun Shen <[email protected]> * Create type checker for geoip function Signed-off-by: Yuanchun Shen <[email protected]> * Define type checker for grok and item operators Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> Signed-off-by: Yuanchun Shen <[email protected]> Co-authored-by: Heng Qian <[email protected]>

Signed-off-by: xinyual <[email protected]>

currantw and others added 30 commits June 1, 2025 13:50

Add CIDR function to PPL (opensearch-project#3036) (opensearch-projec…

3c6ba76

…t#3110) Signed-off-by: currantw <[email protected]>

Add FILLNULL command in PPL (opensearch-project#3032) (opensearch-pro…

6b0a0f3

…ject#3075) Signed-off-by: Norman Jordan <[email protected]> Signed-off-by: normanj-bitquill <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]>

Add trendline PPL command (opensearch-project#3071)

77fe0c2

* Add trendline (With SWA) PPL command --------- Signed-off-by: James Duong <[email protected]> Signed-off-by: Andrew Carbonetto <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]>

Followup: Register OpenSearchTypeSystem to OpenSearchTypeFactory (ope…

82b074e

…nsearch-project#3349) Signed-off-by: Lantao Jin <[email protected]>

Implement ppl join command with Calcite (opensearch-project#3364)

459f00e

Signed-off-by: Lantao Jin <[email protected]>

Enable push down optimization by default (opensearch-project#3366)

ff52d8b

Signed-off-by: Heng Qian <[email protected]>

Implement ppl IN subquery command with Calcite (opensearch-project#…

5f22078

…3371)

fix java 21 syntax

4960a88

Signed-off-by: xinyual <[email protected]>

fix IT for syntax 21

4e45c83

Signed-off-by: xinyual <[email protected]>

[BugFix] Fix the join condition resolving bug introduced by IN subque…

9fb1b64

…ry impl (opensearch-project#3377)

Implement ppl relation subquery command with Calcite (opensearch-proj…

c4af931

…ect#3378) Signed-off-by: Lantao Jin <[email protected]>

Implement ppl exists subquery command with Calcite (opensearch-projec…

824b4d8

…t#3388)

Support multiple table and index pattern (opensearch-project#3409)

5fae05b

* Support multiple table and index pattern Signed-off-by: Heng Qian <[email protected]> * Fix UT Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]>

Separate with/without pushdown ITs (opensearch-project#3413)

bcb89b1

Signed-off-by: Lantao Jin <[email protected]>

yuancu and others added 30 commits June 3, 2025 15:56

fix java 21 syntax and apply spotless on develop code

cc275df

Signed-off-by: xinyual <[email protected]>

fix syntax for IT

a9363ed

Signed-off-by: xinyual <[email protected]>

restore useless change

ebceb03

Signed-off-by: xinyual <[email protected]>

fix IT

55c7ad7

Signed-off-by: xinyual <[email protected]>

remove useless change

eaeadff

Signed-off-by: xinyual <[email protected]>

fix IT

f5b7a7f

Signed-off-by: xinyual <[email protected]>

Revert "Switch percentile implementation to MergingDigest to align wi…

48a8f16

…th OpenSearch (opensearch-project#3698)" This reverts commit 0defdcd.

fix IT

9cc8bdf

Signed-off-by: xinyual <[email protected]>

fix part of IT

40de8d5

Signed-off-by: xinyual <[email protected]>

Support GEO_POINT type (opensearch-project#3511)

0be4e15

--------- Signed-off-by: Heng Qian <[email protected]>

fix IT

46bef64

Signed-off-by: xinyual <[email protected]>

fix IT by add timestamp value

ba0b695

Signed-off-by: xinyual <[email protected]>

revert change

34ed823

Signed-off-by: xinyual <[email protected]>

ignore Prometheus with calcite as previous

dc32437

Signed-off-by: xinyual <[email protected]>

fix Patterns IT

83f51a2

Signed-off-by: xinyual <[email protected]>

fix UT

358d430

Signed-off-by: xinyual <[email protected]>

fix IT by workaround

6c7a2b6

Signed-off-by: xinyual <[email protected]>

fix IT and doctest

d7904aa

Signed-off-by: xinyual <[email protected]>

fix UT and doctest

dda896d

Signed-off-by: xinyual <[email protected]>

fix UT and revert change

eb3e918

Signed-off-by: xinyual <[email protected]>

remove useless UT

00f46b5

Signed-off-by: xinyual <[email protected]>

Support appendcol command with Calcite (opensearch-project#3729)

0979762

Support decimal literal with Calcite (opensearch-project#3673)

169bd70

fix IT

001d277

Signed-off-by: xinyual <[email protected]>

fix bwc

c5875f9

Signed-off-by: xinyual <[email protected]>

fix 21 syntax

ae31cde

Signed-off-by: xinyual <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Backport work around #3744

Backport work around #3744

xinyual commented Jun 6, 2025

Uh oh!

Uh oh!

Backport work around #3744

Are you sure you want to change the base?

Backport work around #3744

Conversation

xinyual commented Jun 6, 2025

Description

Related Issues

Check List

Uh oh!

Uh oh!