Sensors disappearing from the map with sensor communication interruption

[RikDrabs]

With instable sensor communication, on WiFi but more often on LoRaWan links, the indication of the sensor on the map disappears for several minutes, even tens of minutes, before being shown again.
Can the last measurement of the sensor not be held on the map, even if that measurement was ten minutes ago, so that with instable communication links the sensor can be seen permanently on the map ?

[ricki-z]

This is a matter of saved data for this timespan. At the moment we receive 17.500 values per minute. So for the 5 minute interval we use at the moment the database has to retrieve 87.500 values. This needs around 10 seconds. With a longer timespan we would also need more time for retrieval. But we need to do this in under a minute. So we have the choice, more sensors (giving more values per minute) or a longer timespan for the map.
With the 5 minutes we should receive two data transmissions of every sensor. So with a single missing transmission the sensor won't disappear.

[RikDrabs]

All depends how you implement this memory. You don't need to read all data 5 minutes back. If you read valid data, then stop reading back more data for that sensor. So instead of reading 87500 values in normal operation, you could go by at only reading 17500 values, and only reading back when one is missing. And then you could go back for more than 5-10-15-20 minutes, and still be faster than now with the 87500 reads. This probably necessitates another algorithm reading the data back, and could possibly be a huge work, a risk for bugs, and also the risk that this other method has unforeseen consequences on other interactions, other communications - json ? Forget i asked, if too difficult to implement. Otherwise, this would be a nice job, that could alleviate the workload on the server, and could make map access faster than today. This map access is now visibly slowing down because of the high number of connected sensors ... Thanks for keeping on the good work as volunteers!

[ricki-z]

We need to read all data 5 minutes back. This is the interval all clients are expecting. And looking for missing sensors in that data and going back for these isn't really less work. The software would need to go through all received values a second time to check which sensors are missing.
The database backend also would need to work with a most of the data for the choosen interval. Or how should the database decide if the row needs to be transmitted or not?

For the map we use optimized JSON files. They contain only the average values. So the amount of data is growing linear with the count of sensors. In most cases its the Javascript interpreter that has a problem with the amount of values. Even changing the format of the JSON file or converting it to a smaller "format" (i.e. CVS with a header line with the value titles). won't speed up the map.

[RikDrabs]

OK, It was only a suggestion for an improvement.
Again, thanks for the good work. Keep on guys ...
I will close this issue.

[ricki-z]

We would like to create a new API version, that should be more flexible (mobile sensors) and should have a simpler database design. So suggestions are welcome.