[WIP] Improvements to support scale environment 2 #244
+5,706
−1,439
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ode into wip-scale2
…n loading Instrument the loadUnminedTransactions function with detailed Prometheus metrics to track performance and identify bottlenecks during startup: - UTXO index readiness tracking (gauge + wait duration histogram) - Iterator creation timing with success/error status - Iterator processing with detailed transaction statistics (skipped, already mined, locked, added) - Mark transactions on longest chain timing - Sort transactions timing bucketed by volume (<1k, 1k-10k, 10k-100k, 100k-1M, >1M) - Parent chain validation timing and filtered count - AddDirectly calls with individual and batch timing Also fixed unused import in pkg/k8sresolver/k8s.go
Restructured UTracer.Start() to process logging and metrics independently of tracing state. Previously, early return when tracing was disabled prevented log messages, metrics (histograms/ counters), and stats from being recorded. Changes: - Move option processing before tracing check - Execute logging at span start/end regardless of tracing state - Record metrics (histogram/counter) in endFn unconditionally - Gate OpenTelemetry span operations behind tracingEnabled check - Add test coverage for disabled tracing with logging/metrics This allows observability through logs and metrics even when distributed tracing is disabled for performance reasons.
…ode into wip-scale2
When getminingcandidate is called while processNewBlockAnnouncement is processing a new block, immediately return an empty block template for the new height instead of timing out or blocking. This prevents miners from wasting hashrate on stale work and eliminates timeout errors during block processing.
Simplified the mining candidate caching from ~200 lines to ~100 lines by removing: - Infinite loop with multiple continue statements - Duplicated "stale cache" logic (appeared twice) - Complex lock upgrade pattern with double-checking - generationChan coordination mechanism - Unnecessary retry loops New approach: 1. Check cache with read lock (fast path) 2. Acquire write lock for generation (prevents concurrent generation) 3. Double-check cache after acquiring write lock (race prevention) 4. Generate and update cache This maintains all functionality while being much easier to understand and maintain.
Added two new Prometheus metrics to track BlockAssembler state changes:
1. teranode_blockassembly_state_transitions_total{from, to}
- Counter tracking every state transition
- Labeled with from/to states
- Shows transition patterns and frequency
2. teranode_blockassembly_state_duration_seconds{state}
- Histogram of time spent in each state
- Buckets: 1ms to 60s
- Enables P50/P95/P99 duration analysis
These metrics capture state changes between Prometheus scrapes, providing complete visibility into state transitions and durations for Grafana dashboards.
…king Added comprehensive Grafana dashboard to visualize BlockAssembler state transitions and durations: Panels: - State Timeline: Visual timeline showing state changes over time - State Durations: P50/P95/P99 percentiles for each state - State Transitions: Rate of state changes per second - Current State: Real-time state display with color coding - Time in Current State: Duration gauge with thresholds - State Distribution: Pie chart showing time percentage per state - State Entries: Count of state entries in last 5 minutes - State Transition Matrix: Table view of all transitions Enables monitoring of: - Block processing performance (BlockchainSubscription duration) - Mining candidate generation speed (GetMiningCandidate duration) - Reorg frequency and duration - State transition patterns Includes documentation with example queries, alerting rules, and use cases.
Added missing 'name' field in YAML frontmatter to fix agent parse error.
…sion Reverts the GetMiningCandidate caching refactor from b72e399 which caused TestGetBlockAssemblyBlockCandidate/with_10_txs to timeout. The simplified locking approach introduced a deadlock when transactions were added. Restores the original iterative caching logic with generating flag and generationChan coordination which properly handles concurrent requests. Also fixes: - Test data to use proper TxInpoints with parent transaction hashes - Mock expectations for TxCount/QueueLength/SubtreeCount methods - Test cleanup to prevent logger races All blockassembly tests now pass with race detection enabled.
… using filter expressions and separate modules for SetMined
… with filter expressions
…d multiple blocks" This reverts commit 8010f49.
… cannot be returned
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a work in progress branch where developers can quickly iterate on possibly improvements to the scaling environment.
Once we've found good improvements, all or some of these changes will be pushed back to main codebase in seperate PRs.
❗ Do not merge this PR
❗ Do not pull main into this branch