Jamesmoore/arch 350 add statsd option to silo #114
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Jimmy Moore <[email protected]>
Signed-off-by: Jimmy Moore <[email protected]>
Signed-off-by: Jimmy Moore <[email protected]>
Signed-off-by: Jimmy Moore <[email protected]>
Signed-off-by: Jimmy Moore <[email protected]>
Signed-off-by: Jimmy Moore <[email protected]>
Signed-off-by: Jimmy Moore <[email protected]>
…f no dirty blocks Signed-off-by: Jimmy Moore <[email protected]>
Signed-off-by: Jimmy Moore <[email protected]>
Signed-off-by: Jimmy Moore <[email protected]>
Signed-off-by: Jimmy Moore <[email protected]>
Signed-off-by: Jimmy Moore <[email protected]>
Signed-off-by: Jimmy Moore <[email protected]>
Signed-off-by: Jimmy Moore <[email protected]>
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
Comment on lines
+3
to
+19
| import ( | ||
| "context" | ||
| "fmt" | ||
| "sync" | ||
| "time" | ||
|
|
||
| "github.com/loopholelabs/silo/pkg/storage/dirtytracker" | ||
| "github.com/loopholelabs/silo/pkg/storage/expose" | ||
| "github.com/loopholelabs/silo/pkg/storage/migrator" | ||
| "github.com/loopholelabs/silo/pkg/storage/modules" | ||
| "github.com/loopholelabs/silo/pkg/storage/protocol" | ||
| "github.com/loopholelabs/silo/pkg/storage/sources" | ||
| "github.com/loopholelabs/silo/pkg/storage/volatilitymonitor" | ||
| "github.com/loopholelabs/silo/pkg/storage/waitingcache" | ||
|
|
||
| "github.com/smira/go-statsd" | ||
| ) |
There was a problem hiding this comment.
The import block should follow the standard Go import grouping convention:
- Standard library imports
- External third-party packages
- Internal project packages
Please reorganize the imports into these three distinct groups, with a blank line separating each group:
import (
"context"
"fmt"
"sync"
"time"
"github.com/smira/go-statsd"
"github.com/loopholelabs/silo/pkg/storage/dirtytracker"
"github.com/loopholelabs/silo/pkg/storage/expose"
"github.com/loopholelabs/silo/pkg/storage/migrator"
"github.com/loopholelabs/silo/pkg/storage/modules"
"github.com/loopholelabs/silo/pkg/storage/protocol"
"github.com/loopholelabs/silo/pkg/storage/sources"
"github.com/loopholelabs/silo/pkg/storage/volatilitymonitor"
"github.com/loopholelabs/silo/pkg/storage/waitingcache"
)
Suggested change
| import ( | |
| "context" | |
| "fmt" | |
| "sync" | |
| "time" | |
| "github.com/loopholelabs/silo/pkg/storage/dirtytracker" | |
| "github.com/loopholelabs/silo/pkg/storage/expose" | |
| "github.com/loopholelabs/silo/pkg/storage/migrator" | |
| "github.com/loopholelabs/silo/pkg/storage/modules" | |
| "github.com/loopholelabs/silo/pkg/storage/protocol" | |
| "github.com/loopholelabs/silo/pkg/storage/sources" | |
| "github.com/loopholelabs/silo/pkg/storage/volatilitymonitor" | |
| "github.com/loopholelabs/silo/pkg/storage/waitingcache" | |
| "github.com/smira/go-statsd" | |
| ) | |
| import ( | |
| "context" | |
| "fmt" | |
| "sync" | |
| "time" | |
| "github.com/loopholelabs/silo/pkg/storage/dirtytracker" | |
| "github.com/loopholelabs/silo/pkg/storage/expose" | |
| "github.com/loopholelabs/silo/pkg/storage/migrator" | |
| "github.com/loopholelabs/silo/pkg/storage/modules" | |
| "github.com/loopholelabs/silo/pkg/storage/protocol" | |
| "github.com/loopholelabs/silo/pkg/storage/sources" | |
| "github.com/loopholelabs/silo/pkg/storage/volatilitymonitor" | |
| "github.com/loopholelabs/silo/pkg/storage/waitingcache" | |
| "github.com/smira/go-statsd" | |
| ) | |
Spotted by Diamond (based on custom rules)
Is this helpful? React 👍 or 👎 to let us know.
Comment on lines
+50
to
+518
| } | ||
|
|
||
| func (m *Metrics) RemoveVolatilityMonitor(id string, name string) { | ||
| m.remove(m.config.SubVolatilityMonitor, id, name) | ||
| } | ||
|
|
||
| func (m *Metrics) AddMetrics(id string, name string, mm *modules.Metrics) { | ||
| lastmet := &modules.MetricsSnapshot{ | ||
| ReadOpsSize: make(map[int]uint64), | ||
| WriteOpsSize: make(map[int]uint64), | ||
| } | ||
|
|
||
| m.add(m.config.SubMetrics, id, name, m.config.TickMetrics, func() { | ||
| met := mm.GetMetrics() | ||
| m.updateMetric(id, name, m.config.SubMetrics, "read_ops", lastmet.ReadOps, met.ReadOps) | ||
| m.updateMetric(id, name, m.config.SubMetrics, "read_bytes", lastmet.ReadBytes, met.ReadBytes) | ||
| m.updateMetric(id, name, m.config.SubMetrics, "read_errors", lastmet.ReadErrors, met.ReadErrors) | ||
| m.updateMetric(id, name, m.config.SubMetrics, "read_time", lastmet.ReadTime, met.ReadTime) | ||
| m.updateMetric(id, name, m.config.SubMetrics, "write_ops", lastmet.WriteOps, met.WriteOps) | ||
| m.updateMetric(id, name, m.config.SubMetrics, "write_bytes", lastmet.WriteBytes, met.WriteBytes) | ||
| m.updateMetric(id, name, m.config.SubMetrics, "write_errors", lastmet.WriteErrors, met.WriteErrors) | ||
| m.updateMetric(id, name, m.config.SubMetrics, "write_time", lastmet.WriteTime, met.WriteTime) | ||
| m.updateMetric(id, name, m.config.SubMetrics, "flush_ops", lastmet.FlushOps, met.FlushOps) | ||
| m.updateMetric(id, name, m.config.SubMetrics, "flush_errors", lastmet.FlushErrors, met.FlushErrors) | ||
| m.updateMetric(id, name, m.config.SubMetrics, "flush_time", lastmet.FlushTime, met.FlushTime) | ||
|
|
||
| // Add the size metrics here... | ||
| for _, v := range modules.OpSizeBuckets { | ||
| m.updateMetric(id, name, m.config.SubMetrics, fmt.Sprintf("read_ops_size_%d", v), lastmet.ReadOpsSize[v], met.ReadOpsSize[v]) | ||
| m.updateMetric(id, name, m.config.SubMetrics, fmt.Sprintf("write_ops_size_%d", v), lastmet.WriteOpsSize[v], met.WriteOpsSize[v]) | ||
| } | ||
| lastmet = met | ||
| }) | ||
| } | ||
|
|
||
| func (m *Metrics) RemoveMetrics(id string, name string) { | ||
| m.remove(m.config.SubMetrics, id, name) | ||
| } | ||
|
|
||
| func (m *Metrics) AddNBD(id string, name string, mm *expose.ExposedStorageNBDNL) { | ||
| lastmet := &expose.DispatchMetrics{} | ||
|
|
||
| m.add(m.config.SubNBD, id, name, m.config.TickNBD, func() { | ||
| met := mm.GetMetrics() | ||
| m.updateMetric(id, name, m.config.SubNBD, "packets_in", lastmet.PacketsIn, met.PacketsIn) | ||
| m.updateMetric(id, name, m.config.SubNBD, "packets_out", lastmet.PacketsOut, met.PacketsOut) | ||
| m.updateMetric(id, name, m.config.SubNBD, "read_at", lastmet.ReadAt, met.ReadAt) | ||
| m.updateMetric(id, name, m.config.SubNBD, "read_at_bytes", lastmet.ReadAtBytes, met.ReadAtBytes) | ||
| m.updateMetric(id, name, m.config.SubNBD, "read_at_time", uint64(lastmet.ReadAtTime), uint64(met.ReadAtTime)) | ||
| m.updateMetric(id, name, m.config.SubNBD, "active_reads", lastmet.ActiveReads, met.ActiveReads) | ||
| m.updateMetric(id, name, m.config.SubNBD, "write_at", lastmet.WriteAt, met.WriteAt) | ||
| m.updateMetric(id, name, m.config.SubNBD, "write_at_bytes", lastmet.WriteAtBytes, met.WriteAtBytes) | ||
| m.updateMetric(id, name, m.config.SubNBD, "write_at_time", uint64(lastmet.WriteAtTime), uint64(met.WriteAtTime)) | ||
| m.updateMetric(id, name, m.config.SubNBD, "active_writes", lastmet.ActiveWrites, met.ActiveWrites) | ||
| lastmet = met | ||
| }) | ||
| } | ||
| func (m *Metrics) RemoveNBD(id string, name string) { | ||
| m.remove(m.config.SubNBD, id, name) | ||
| } | ||
|
|
||
| func (m *Metrics) AddWaitingCache(id string, name string, wc *waitingcache.Remote) { | ||
| lastmet := &waitingcache.Metrics{} | ||
| m.add(m.config.SubWaitingCache, id, name, m.config.TickWaitingCache, func() { | ||
| met := wc.GetMetrics() | ||
| m.updateMetric(id, name, m.config.SubWaitingCache, "waiting_for_block", lastmet.WaitForBlock, met.WaitForBlock) | ||
| m.updateMetric(id, name, m.config.SubWaitingCache, "waiting_for_block_had_remote", lastmet.WaitForBlockHadRemote, met.WaitForBlockHadRemote) | ||
| m.updateMetric(id, name, m.config.SubWaitingCache, "waiting_for_block_had_local", lastmet.WaitForBlockHadLocal, met.WaitForBlockHadLocal) | ||
| m.updateMetric(id, name, m.config.SubWaitingCache, "waiting_for_block_lock", lastmet.WaitForBlockLock, met.WaitForBlockLock) | ||
| m.updateMetric(id, name, m.config.SubWaitingCache, "waiting_for_block_lock_done", lastmet.WaitForBlockLockDone, met.WaitForBlockLockDone) | ||
| m.updateMetric(id, name, m.config.SubWaitingCache, "mark_available_local_block", lastmet.MarkAvailableLocalBlock, met.MarkAvailableLocalBlock) | ||
| m.updateMetric(id, name, m.config.SubWaitingCache, "mark_available_remote_block", lastmet.MarkAvailableRemoteBlock, met.MarkAvailableRemoteBlock) | ||
| m.updateMetric(id, name, m.config.SubWaitingCache, "available_local", lastmet.AvailableLocal, met.AvailableLocal) | ||
| m.updateMetric(id, name, m.config.SubWaitingCache, "available_remote", lastmet.AvailableRemote, met.AvailableRemote) | ||
| lastmet = met | ||
| }) | ||
|
|
||
| } | ||
| func (m *Metrics) RemoveWaitingCache(id string, name string) { | ||
| m.remove(m.config.SubWaitingCache, id, name) | ||
| } | ||
|
|
||
| func (m *Metrics) AddCopyOnWrite(id string, name string, cow *modules.CopyOnWrite) { | ||
| lastmet := &modules.CopyOnWriteMetrics{} | ||
| m.add(m.config.SubCopyOnWrite, id, name, m.config.TickCopyOnWrite, func() { | ||
| met := cow.GetMetrics() | ||
| m.updateMetric(id, name, m.config.SubCopyOnWrite, "size", lastmet.MetricSize, met.MetricSize) | ||
| m.updateMetric(id, name, m.config.SubCopyOnWrite, "nonzero_size", lastmet.MetricNonZeroSize, met.MetricNonZeroSize) | ||
| m.updateMetric(id, name, m.config.SubCopyOnWrite, "overlay_size", lastmet.MetricOverlaySize, met.MetricOverlaySize) | ||
| m.updateMetric(id, name, m.config.SubCopyOnWrite, "zero_read_ops", lastmet.MetricZeroReadOps, met.MetricZeroReadOps) | ||
| m.updateMetric(id, name, m.config.SubCopyOnWrite, "zero_read_bytes", lastmet.MetricZeroReadBytes, met.MetricZeroReadBytes) | ||
| m.updateMetric(id, name, m.config.SubCopyOnWrite, "zero_pre_write_read_ops", lastmet.MetricZeroPreWriteReadOps, met.MetricZeroPreWriteReadOps) | ||
| m.updateMetric(id, name, m.config.SubCopyOnWrite, "zero_pre_write_read_bytes", lastmet.MetricZeroPreWriteReadBytes, met.MetricZeroPreWriteReadBytes) | ||
| lastmet = met | ||
| }) | ||
| } | ||
|
|
||
| func (m *Metrics) RemoveCopyOnWrite(id string, name string) { | ||
| m.remove(m.config.SubCopyOnWrite, id, name) | ||
| } |
There was a problem hiding this comment.
The exported types and functions in this package lack documentation comments. Each exported element (such as DefaultConfig, New, Metrics struct, and methods like AddSyncer, RemoveSyncer, etc.) should have a preceding comment that explains its purpose, behavior, and usage. This is particularly important for a metrics package that will be integrated with external systems. Adding these comments would improve code maintainability and make it easier for other developers to understand how to properly use this statsd implementation.
Spotted by Diamond (based on custom rules)
Is this helpful? React 👍 or 👎 to let us know.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the option to export stats via statsd.
The internal architecture of metrics in Silo is pull based - we ask components to snapshot their metrics.
The statsd module then sends any changes to the destination.
Also adds a quick fix on dirty tracker to avoid max age metric being volatile.
Adds docker compose config for Vector setup to test statsd metrics.