Add earliest and latest in condition function #3640

xinyual · 2025-05-20T06:15:35Z

Description

This pr add latest/earliest as two condition function, it will calculate the relative time according to relative string and current time.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#3639

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: xinyual <[email protected]>

penghuo · 2025-05-20T17:27:26Z

update docs please.
what is expectation of earliest and latest? Do we support it in where command?

dai-chen

In SPL doc, these functions are applied to _raw field, but in our impl it's just an alias of min/max?

Signed-off-by: xinyual <[email protected]>

xinyual · 2025-05-22T08:05:02Z

update docs please.

what is expectation of earliest and latest? Do we support it in where command?

Add the doc.
This PR only add them as the aggregation. Will raise another PR as the boolean function.

Signed-off-by: xinyual <[email protected]>

LantaoJin · 2025-05-30T01:36:00Z

integ-test/src/test/java/org/opensearch/sql/calcite/standalone/CalcitePPLEventstatsIT.java

+            "1970-01-18 20:22:32",
+            "2018-08-19 00:00:00"));


I am afraid the current implementation may be wrong, the current implementation treat the earliest and latest as window function with default window frame. IMO, the behavior is odd. Please check again if they require to use other window frame instead default.

@xinyual does spark-ppl support eventstats earliest(birthdate), latest(birthdate), if not, can we decide later, and only support it as condition function?
From user perspective, It is alias of min/max function, not impact usability.

Spark-ppl only supports it as condition function. Currently I forbid them as window/aggregation in the latest code.

penghuo · 2025-06-04T17:39:57Z

docs/user/ppl/functions/condition.rst

+
+Usage: earliest(relative_string, field) returns true if the field is after transferring the relative_string.
+
+relative_string: 


Does relative_string follow RELATIVE_TIMESTAMP or OpenSearch date math?

relative_string should include a clear definition of the supported time modifiers.

Does relative_string follow RELATIVE_TIMESTAMP or OpenSearch date math?

Yes.

relative_string should include a clear definition of the supported time modifiers.

@xinyual add link to https://github.com/opensearch-project/opensearch-spark/blob/main/docs/ppl-lang/functions/ppl-datetime.md#relative_timestamp here.

docs/user/ppl/functions/condition.rst

penghuo · 2025-06-04T18:39:30Z

core/src/main/java/org/opensearch/sql/expression/function/udf/condition/LatestFunction.java

+    ZonedDateTime candidateDatetime = ZonedDateTime.ofInstant(candidate, clock.getZone());
+    ZonedDateTime latest =
+        getRelativeZonedDateTime(
+            expression, ZonedDateTime.ofInstant(clock.instant(), clock.getZone()));
+    return latest.isAfter(candidateDatetime);


In case all timestamp use same timezone, this PR actually does not have similar issue as #3725?

Signed-off-by: xinyual <[email protected]>

LantaoJin

LGTM once doc comment addressed.

LantaoJin · 2025-06-07T03:30:36Z

core/src/main/java/org/opensearch/sql/calcite/utils/PPLOperandTypes.java

@@ -61,4 +61,9 @@ private PPLOperandTypes() {}
  public static final UDFOperandMetadata DATE_OR_TIMESTAMP_OR_STRING =
      UDFOperandMetadata.wrap(
          (CompositeOperandTypeChecker) OperandTypes.DATE_OR_TIMESTAMP.or(OperandTypes.STRING));
+  public static final UDFOperandMetadata STRING_DATE_OR_TIMESTAMP =


minor: change to STRING_OR_TIMESTAMP

Already rename to STRING_TIMESTMAP, which means accept (STRING, TIMESTAMP)

LantaoJin · 2025-06-07T03:34:00Z

docs/user/ppl/functions/condition.rst

+
+Usage: earliest(relative_string, field) returns true if the field is after transferring the relative_string.
+
+relative_string: 


Does relative_string follow RELATIVE_TIMESTAMP or OpenSearch date math?

Yes.

relative_string should include a clear definition of the supported time modifiers.

@xinyual add link to https://github.com/opensearch-project/opensearch-spark/blob/main/docs/ppl-lang/functions/ppl-datetime.md#relative_timestamp here.

Signed-off-by: xinyual <[email protected]>

xinyual added 4 commits May 20, 2025 13:39

add earliest and latest

7d2813d

Signed-off-by: xinyual <[email protected]>

add more ITs

4a82bd5

Signed-off-by: xinyual <[email protected]>

reuse max/min

5589557

Signed-off-by: xinyual <[email protected]>

revert useless change

b0246c4

Signed-off-by: xinyual <[email protected]>

xinyual marked this pull request as ready for review May 20, 2025 06:15

xinyual requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, mengweieric, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto, GumpacG, ykmr1224, LantaoJin, noCharger and qianheng-aws as code owners May 20, 2025 06:15

dai-chen reviewed May 20, 2025

View reviewed changes

LantaoJin added the calcite calcite migration releated label May 22, 2025

add doc

6fe9895

Signed-off-by: xinyual <[email protected]>

xinyual added 7 commits May 29, 2025 17:39

change doc

343cc4e

Signed-off-by: xinyual <[email protected]>

update doc

8460647

Signed-off-by: xinyual <[email protected]>

update doc

591e4a1

Signed-off-by: xinyual <[email protected]>

update doc

c71a9b6

Signed-off-by: xinyual <[email protected]>

add more doc

a5309ec

Signed-off-by: xinyual <[email protected]>

update to utc_timestamp

c280092

Signed-off-by: xinyual <[email protected]>

remove useless change

4b9ad84

Signed-off-by: xinyual <[email protected]>

LantaoJin requested changes May 30, 2025

View reviewed changes

Merge remote-tracking branch 'origin/main' into addEarliest

49e9657

penghuo reviewed Jun 4, 2025

View reviewed changes

xinyual added 8 commits June 5, 2025 17:13

update the doc

966ed23

Signed-off-by: xinyual <[email protected]>

merge from main

3f14025

Signed-off-by: xinyual <[email protected]>

forbidden the earliest/latest in aggregation/window function

726107c

Signed-off-by: xinyual <[email protected]>

forbid IT

837a4d4

Signed-off-by: xinyual <[email protected]>

revert doc

7dc2e88

Signed-off-by: xinyual <[email protected]>

merge from main

baa5ee4

Signed-off-by: xinyual <[email protected]>

add type checker

c5a8eea

Signed-off-by: xinyual <[email protected]>

revert useless change

c986508

Signed-off-by: xinyual <[email protected]>

LantaoJin previously approved these changes Jun 7, 2025

View reviewed changes

xinyual added 2 commits June 9, 2025 11:32

add doc

853ca1a

Signed-off-by: xinyual <[email protected]>

add link

68d40bd

Signed-off-by: xinyual <[email protected]>

xinyual dismissed LantaoJin’s stale review via 68d40bd June 9, 2025 05:11

LantaoJin previously approved these changes Jun 9, 2025

View reviewed changes

xinyual added 2 commits June 10, 2025 11:00

merge from main

883fa78

Signed-off-by: xinyual <[email protected]>

apply spotless

0caff79

Signed-off-by: xinyual <[email protected]>

xinyual dismissed LantaoJin’s stale review via 0caff79 June 10, 2025 03:08

LantaoJin approved these changes Jun 10, 2025

View reviewed changes

xinyual changed the title ~~Add earliest and latest in aggregation and window function~~ Add earliest and latest in condition function Jun 10, 2025

qianheng-aws approved these changes Jun 10, 2025

View reviewed changes

qianheng-aws merged commit 246138c into opensearch-project:main Jun 10, 2025
22 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add earliest and latest in condition function #3640

Add earliest and latest in condition function #3640

xinyual commented May 20, 2025 •

edited

Loading

Uh oh!

penghuo commented May 20, 2025

Uh oh!

dai-chen left a comment

Uh oh!

xinyual commented May 22, 2025

Uh oh!

LantaoJin May 30, 2025

Uh oh!

penghuo Jun 5, 2025

Uh oh!

xinyual Jun 6, 2025

Uh oh!

penghuo Jun 4, 2025

Uh oh!

LantaoJin Jun 7, 2025

Uh oh!

Uh oh!

penghuo Jun 4, 2025

Uh oh!

xinyual Jun 5, 2025

Uh oh!

LantaoJin left a comment

Uh oh!

LantaoJin Jun 7, 2025

Uh oh!

xinyual Jun 9, 2025

Uh oh!

LantaoJin Jun 7, 2025

Uh oh!

Uh oh!

Uh oh!


		Usage: earliest(relative_string, field) returns true if the field is after transferring the relative_string.

		relative_string:

Add earliest and latest in condition function #3640

Add earliest and latest in condition function #3640

Conversation

xinyual commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

penghuo commented May 20, 2025

Uh oh!

dai-chen left a comment

Choose a reason for hiding this comment

Uh oh!

xinyual commented May 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LantaoJin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

xinyual commented May 20, 2025 •

edited

Loading