Fix skipme handling #13244
Fix skipme handling #13244
Conversation
Signed-off-by: Monica Sekoyan <[email protected]>
Signed-off-by: monica-sekoyan <[email protected]>
| if len(cuts) > 1: | ||
| # Before we mix the datasets in the config, we filter cutsets to exclude cuts | ||
| # with valid "_skipme" values to mix the data correctly. | ||
| cuts = [cut.filter(PlaceholderFilter()) for cut in cuts] |
There was a problem hiding this comment.
is this necessary? Don;t samples with _skipme already get filtered in read_cutset_from_config function with PlaceholderFilter there ?
There was a problem hiding this comment.
we need to filter before cuts will get mixed with the specified weights, because if we mix n samples vs n - n_skipme_samples, resulted cutset will be different
There was a problem hiding this comment.
in read_cutset_from_config we do the filtering for non-merged cutsets, but I think I can change the place of the filtering to do one for both cases
nithinraok
added
skip-linting
Run CICD
r2.3.0
|
[🤖]: Hi @monica-sekoyan 👋, We wanted to let you know that a CICD pipeline for this PR just finished successfully So it might be time to merge this PR or get some approvals I'm just a bot so I'll leave it you what to do next. //cc @pablo-garay @ko3n1g |
|
@nithinraok the CI tests passed. Please review the PR when you get a chance. |
Signed-off-by: Monica Sekoyan <[email protected]>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Signed-off-by: Monica Sekoyan <[email protected]>
Signed-off-by: monica-sekoyan <[email protected]>
|
@pzelasko @nithinraok what do you think about this new implementation? |
|
I stopped the CI pipeline since there was a unit test failure already. Please take a look at that and also probably best to get a review on the latest chagnes before retrying again. |
Important
The
Update branchbutton must only be pressed in very rare occassions.An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.
What does this PR do ?
Skipme filtering was not handled before for the cases when data was provided with
input_cfgto Lhotse Dataloader config.Collection: [Note which collection this PR will affect]
Changelog
Usage
# Add a code snippet demonstrating how to use thisGitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information