About Dashboard repsonse time too long!

[yangshike]

hello every one,
I have been using hyperdx to create a dashboard recently,
My original log was nginx, with 20 million logs per day. When aggregating URL fields to query party data for one day, the dashboard response was very slow. I looked at it:

1. Single indicator query returns over 500000 records to browser (10MB)
2. Aggregation did not include Lily, but returned all records, including a large number of records with count() set to 1
3. The aggregated field has no alias, resulting in long names and incomplete dashboard display
   Thank you.

[yangshike]

[yangshike]

Another issue is that when aggregating nginx request_url fields, inaccurate data often occurs, and the most visited request_url does not appear in the aggregation chart

[yangshike]

The same query yields different results twice:

[knudtty]

Hey yangshike,

Your screenshot of the json shows 500,000 lines, but each row takes up 5 lines so it's really 100,000 rows. We actually have a limit here which is 100,000 because as you can see it really does take that long to render everything. That limit is probably what your query is running into. It would also explain why you run the same query twice but get different results - your total number of results is greater than 100,000 and each query is grabbing from a different subset from inside the total set.

How many different URLs do you have? The chart typically renders 60-100ish different points, so to hit the 100,000 limit you'd need at least 1000 different URLs, probably closer to 2000 if you're seeing large differences between charts. Maybe you have a dynamic route (like /edu/eduApp/:serviceId) that has many different instances? Unfortunately, high cardinality data like this is a big workload for the DB and does mean a lot of data streamed down if you want to view on a chart.