Library search broken

[maniac103]

Since 9.0 (I recently upgraded from 8.5) library search is returning flat out wrong data. One example (debug log from an app I wrote doing search requests via cometd):

01-15 10:10:01.850  7572  7614 D CometdClient: --> POST http://192.168.100.10:9000/cometd
01-15 10:10:01.850  7572  7614 D CometdClient: Content-Type: application/json; charset=utf-8
01-15 10:10:01.850  7572  7614 D CometdClient: Content-Length: 243
01-15 10:10:01.850  7572  7614 D CometdClient: User-Agent: Squeezer-squeezer/1.0
01-15 10:10:01.850  7572  7614 D CometdClient: 
01-15 10:10:01.850  7572  7614 D CometdClient: [{"channel":"/slim/request","clientId":"0303b530","data":{"request":["b8:ae:ed:7c:ce:d4",["browselibrary","items","0","300","mode:albums","search:a-ha","menu:browselibrary","useContextMenu:1"]],"response":"/0303b530/slim/request/52"},"id":96}]
01-15 10:10:01.850  7572  7614 D CometdClient: --> END POST (243-byte body)
01-15 10:10:02.036  7572  7614 D CometdClient: <-- 200 OK http://192.168.100.10:9000/cometd (185ms)
01-15 10:10:02.036  7572  7614 D CometdClient: Server: Lyrion Music Server (9.0.1 - 1736238071)
01-15 10:10:02.036  7572  7614 D CometdClient: Cache-Control: no-cache
01-15 10:10:02.036  7572  7614 D CometdClient: Connection: Keep-Alive
01-15 10:10:02.036  7572  7614 D CometdClient: Pragma: no-cache
01-15 10:10:02.036  7572  7614 D CometdClient: Content-Length: 77
01-15 10:10:02.036  7572  7614 D CometdClient: Content-Type: application/json
01-15 10:10:02.036  7572  7614 D CometdClient: Expires: -1
01-15 10:10:02.036  7572  7614 D CometdClient: X-Time-To-Serve: 0.123763084411621
01-15 10:10:02.036  7572  7614 D CometdClient: 
01-15 10:10:02.036  7572  7614 D CometdClient: [{"id":96,"clientId":"0303b530","successful":true,"channel":"/slim/request"}]
01-15 10:10:02.036  7572  7614 D CometdClient: <-- END HTTP (77-byte body)
01-15 10:10:02.472  7572  7621 D CometdClient: Received event message (111450 bytes): Message(channelId=ChannelId(channel=/0303b530/slim/request/52), clientId=null, id=96, successful=true, data={"offset":0,"base":{"actions":{"set-preset-2":{"player":0,"itemsParams":"presetParams","cmd":["jivefavorites","set_preset","key:2"]},"add-hold":{"params":{"menu":1,"cmd":"insert"},"cmd":["playlistcontrol"],"itemsParams":"commonParams","player":0},"set-preset-1":{"player":0,"cmd":["jivefavorites","set_preset","key:1"],"itemsParams":"presetParams"},"playControl":{"window":{"isContextMenu":1},"player":0,"params":{"_index":"0","_quantity":"300","mode":"albums","search":"a-ha","menu":"browselibrary","useContextMenu":"1"},"cmd":["browselibrary","items"],"itemsParams":"playControlParams"},"set-preset-3":{"itemsParams":"presetParams","cmd":["jivefavorites","set_preset","key:3"],"player":0},"set-preset-0":{"itemsParams":"presetParams","cmd":["jivefavorites","set_preset","key:0"],"player":0},"set-preset-4":{"player":0,"itemsParams":"presetParams","cmd":["jivefavorites","set_preset","key:4"]},"set-preset-6":{"player":0,"itemsParams":"presetParams","cmd":["jivefavorites","set_preset","key:6"]},"set-preset-9":{"player":0,"cmd":["jivefavorites","set_preset","key:9"],"itemsParams":"presetParams"},"go":{"player":0,"cmd":["browselibrary","items"],"params":{"menu":1,"mode":"tracks"},"itemsParams":"commonParams"},"set-preset-7":{"player":0,"cmd":["jivefavorites","set_preset","key:7"],"itemsParams":"presetParams"},"play":{"itemsParams":"commonParams","params":{"menu":1,"cmd":"load"},"cmd":["playlistcontrol"],"nextWindow":"nowPlaying","player":0},"set-preset-5":{"cmd":["jivefavorites","set_preset","key:5"],"itemsParams":"presetParams","player":0},"set-preset-8":{"itemsParams":"presetParams","cmd":["jivefavorites","set_preset","key:8"],"player":0},"add":{"player":0,"cmd":["playlistcontrol"],"params":{"menu":1,"cmd":"add"},"itemsParams":"commonParams"},"more":{"params":{"menu":1},"cmd":["albuminfo","items"],"itemsParams":"commonParams","window":{"isContextMenu":1},"player":0}}},"item_loop":[{"commonParams":{"album_id":7629,"performance":""},"type":"playlist","text":"Das Phantom der Oper: Die Höhepunkte der Hamburger Aufführung\nAndrew Lloyd Webber","presetParams":{"favorites_type":"playlist","icon":"music/f8f8a285/cover","favorites_title":"Das Phantom der Oper: Die Höhepunkte der Hamburger Aufführung","favorites_url":"db:album.title=Das%20Phantom%20der%20Oper%3A%20Die%20H%C3%B6hepunkte%20der%20Hamburger%20Auff%C3%BChrung&contributor.name=Andrew%20Lloyd%20Webber"},"icon-id":"f8f8a285","icon":"music/f8f8a285/cover"},{"commonParams":{"album_id":7645,"performance":""},"type":"playlist","presetParams":{"favorites_title":"Sister Act 2: Back in the Habit","favorites_url":"db:album.title=Sister%20Act%202%3A%20Back%20in%20the%20Habit&contributor.name=Diverse%20Interpreten","favorites_type":"playlist","icon":"music/5921d1cc/cover"},"text":"Sister Act 2: Back in the Habit\nDiverse Interpreten","icon":"music/5921d1cc/cover","icon-id":"5921d1cc"},{"commonParams":{"performance":"","album_id":7423},"type":"playlist","text":"Café del Mar: 25th Anniversary 1980-2005 (Disk 1 von 3)\nDiverse Interpreten","presetParams":{"icon":"music/94445eb1/cover","favorites_type":"playlist","favorites_url":"db:album.title=Caf%C3%A9%20del%20Mar%3A%2025th%20Anniversary%201980-2005&contributor.name=Diverse%20Interpreten","favorites_title":"Café del Mar: 25th Anniversary 1980-2005"},"icon-id":"94445eb1","icon":"music/94445eb1/cover"},{"type":"playlist","commonParams":{"performance":"","album_id":7469},"icon":"music/22fe2afb/cover","icon-id":"22fe2afb","text":"Best of Chill-Out: Listen and Relax (Disk 2 von 3)\nDiverse Interpreten","presetParams":{"favorites_url":"db:album.title=Best%20of%20Chill-Out%3A%20Listen%20and%20Relax&contributor.name=Diverse%20Interpreten","favorites_title":"Best of Chill-Out: Listen and Relax","icon":"music/22fe2afb/cover","favorites_type":"playlist"}},{"commonParams":{"album_id":7407,"performance":""},"type":"playlist","text":"Der Hauptgewinner\nRüdiger Hoffma

You see I searched for albums matching 'a-ha' and got random results, including Phantom of the Opera or Sister Act 2 OST, which are totally unrelated to a-ha, and aren't tagged as such. With such a search, I'd expect a text search for 'a-ha' in artist or album artist tags, but neither of those albums (obviously) is tagged as such.

Am I missing something here? I can't try at the moment, but am pretty sure 8.5 returned more meaningful search results for such queries.

[maniac103]

BTW, this is not about returning too much data only. When searching other artists in my library, I get an amout of search results that's too small, or no search results at all. It's also not restricted to albums, as I'm seeing the same kind of results for mode:tracks.

Please let me know if I can provide more data to pinpoint this.

[michaelherger]

Please enable logging for database.sql in LMS (Settings/Advanced/Logging) and see what SQL queries your queries result in. Could this be a side-effect of using the full text search vs. not using it? The feature can be enabled/disabled in the plugins settings.

[maniac103]

Well, the 'too much data is returned for a-ha case can be easily explained by the debug log:

[25-01-15 12:54:00.7463] Slim::Plugin::FullTextSearch::Plugin::createHelperTable (294) Fulltext search query (album): INSERT INTO albumsSearch SELECT SUBSTR(fulltext.id, 33) AS id, FULLTEXTWEIGHT(matchinfo(fulltext)) AS fulltextweight FROM fulltext WHERE fulltext MATCH 'type:album a* AND ha*' 

I searched for a-ha, not a ha though ;-)

When disabling full text search plugin (I assume that's what you was referring to above?), I get this query:

[25-01-15 12:57:37.9025] Slim::Control::Queries::albumsQuery (694) Albums totals query: SELECT albums.artwork AS 'albums.artwork', albums.compilation AS 'albums.compilation', albums.contributor AS 'albums.contributor', albums.disc AS 'albums.disc', albums.discc AS 'albums.discc', albums.extid AS 'albums.extid', albums.id AS 'albums.id', albums.title AS 'albums.title', albums.titlesearch AS 'albums.titlesearch', albums.titlesort AS 'albums.titlesort', albums.year AS 'albums.year', contributors.name AS 'contributors.name' FROM albums JOIN contributors ON contributors.id = albums.contributor WHERE (albums.titlesearch LIKE ? OR albums.titlesearch LIKE ?) GROUP BY albums.id ORDER BY albums.titlesort COLLATE de_DE , albums.disc  / ["A HA%", "% A HA%"]

... which yields 0 results, which also isn't intended (FWIW, My music -> Album artists -> a-ha yields 4 results).
Shouldn't / couldn't it also search contributors.namesearch?

[michaelherger]

Oh, you're searching for albums - which would search the album title. You obviously don't have any albums called "a-ha", but albums by an artist of that name. You can't use browselibrary to search for albums by artist using a name search. You'd have to search for the artist first, then tell the albums query for which artist_id you want the albums.

[maniac103]

Wouldn't that kind of search results (albumartist matching a search for albums, artist matching a search for tracks) be a useful addition, though?

Besides that, there's still the issue of full text search being completely useless here. If I provide the search term 'a-ha', why doesn't it do ...MATCH 'type:album a-ha'? Judging by sqlite3, this would provide the expected search result.
(NB: judging by sqlite3 may give a wrong impression though. I tried that by searching for another artist's name, for which the CLI search results provide 0 matches, but a SELECT * FROM fulltext WHERE fulltext MATCH 'type:album schiller' in sqlite3 shows me meaningful results. So there may be more to this, and I may have had fulltext search enabled in 8.5, which would explain the above expectation)

TL,DR: non-full text search probably is OK as in 'has unchanged behavior', but full text search has issues.

[maniac103]

Having scanned through the full text search plugin a bit (which I don't yet fully understand though), I gather that I can fix my a-ha case by quoting (thus sending search term "a-ha"). This gives me the expected results also for the album search (which returns the a-ha albums I have). However, I wonder

• If sending quoted search terms might give problems if full text search is disabled?
• Why sending "a-ha" gives me albums made by a-ha, but sending "Schiller" doesn't give me albums made by Schiller?

[michaelherger]

In the other issue you mentioned following slim browse menus as you receive them from the server. Is this the case for this search, too?

It would indeed be great if the album search allowed you to search for its artist magically. But then the question as always would be: how far would you want to go? Should we also find albums by track lyrics? By soloist? Director? With a particular track name on it?

It all would be possible, and somewhat already is with the help of the full text search plugin. But as you've figured out this can lead to somewhat surprising results - the artist name "A-Ha" is a perfect example of how a name, interpreted loosely, can lead to totally invalid results. And not using full text search, but trying to figure out what the search term might be, or providing specific parameters for all possible search options would lead to a massive increase of the already pretty high complexity, in particular when dealing with contributors and their man different roles.

[maniac103]

It would indeed be great if the album search allowed you to search for its artist magically. But then the question as always would be: how far would you want to go? Should we also find albums by track lyrics? By soloist? Director? With a particular track name on it?

I'd say stop at album artist, but the argument is fair, even though - as mentioned in #1298 - you expected it to work like that when writing 'Music & artist information' as well ;-)
But taking that aside and accepting full text search is required to fully use that plugin, my two questions from here are still valid, I think ... and the second question still smells like a bug somewhere.

[maniac103]

Why sending "a-ha" gives me albums made by a-ha, but sending "Schiller" doesn't give me albums made by Schiller?

I traced down this one to this check triggering. Apparently schiller is considered a popular search term in my library since I happen to have a lot of albums made by Schiller (> 700 tracks), which (from my understanding) enables some auto-constrain mode to avoid long result lists.
It seems like rather surprising behavior to me, though, to have the search results depend on the amount of potential search results. Can you shed some light onto why this is implemented like that? Could we make this (either the check or the value of LARGE_RESULTSET) optional?

[michaelherger]

It's a performance optimisation, as other typing something like "an" or "or" would have killed the server.

It's been a long time since I worked on that... I don't know whether this could be tweaked reasonably, other than by adding yet another preference...