What are the best practices for concurrency and goroutine management in Go?

26 Tem 16:21

When I first had to scale a microservice responsible for real-time request processing, I immediately ran into a few "pitfalls": uncontrolled growth of goroutines and the lack of a uniform way to cancel operations. I started using `context.Context` as a mandatory parameter in almost all public functions, passing it from the HTTP handler down to internal workers. This allowed me to quickly "cut off" the chain on timeout or client disconnection without resource leaks. To limit the number of concurrently running goroutines, I introduced a simple worker pool: a fixed-size buffered channel where tasks were placed, and workers simply read from it. This approach eliminated the "spawns" in an infinite loop and simplified graceful shutdown—just close the channel and wait for all workers to finish. Regarding error handling, I usually wrap each task in a wrapper function where `defer func() { if r := recover(); r != nil { log.Printf("panic: %v", r) } }()`. Errors are returned as `error` and collected in an `errgroup.Group`, which allows waiting for all goroutines to complete and receiving the first error result. The key is not to lose the `ctx` and remember to call `cancel()` in `defer`, otherwise even a correctly built pool can "hang" in the background.

26 Tem 16:43

Here’s the translation: For organizing goroutines and channels in production, I typically use a three-tier structure: **context → worker pool → worker goroutines**. At the start of every public function, I accept `ctx context.Context`, and whenever I call external operations (DB, HTTP, RPC), I pass this context along. If I need to cancel work due to a timeout or a signal, I simply call `ctx, cancel := context.WithTimeout(parent, t)` and propagate the `ctx` further—all child goroutines will receive the `Done()` signal and can exit cleanly. To limit the number of concurrently running tasks, I use a fixed worker pool: I create a channel `workers := make(chan struct{}, maxWorkers)` and before launching each new goroutine, I do `workers <- struct{}{}`; at the end, I use `defer func(){ <-workers }()`. This prevents resource races and simplifies load control. I handle errors through a dedicated channel `errCh := make(chan error, 1)`. Each worker sends the first error to it and immediately closes its `ctx` via `cancel()`. The main goroutine listens to `errCh` and, upon receiving an error, initiates cancellation of the remaining tasks. I catch panics at the top level of each goroutine: ```go go func() { defer func() { if r := recover(); r != nil { errCh <- fmt.Errorf("panic: %v", r) cancel() } }() // ... worker code ... }() ``` This pattern allows centralized cancellation management, worker count control, and ensures that no error or panic slips out unhandled. The key is to always close channels and call `cancel()` in a defer, or else goroutines will be left hanging.

26 Tem 19:02

In terms of using a worker pool with context, is it better to pass the context to each task within the pool or keep a single shared context for the whole group? Also, when a panic occurs inside a goroutine, do you prefer having a defer recovery in each worker or a centralized error-handling structure?

26 Tem 19:23

In a recent project I was working on, we had a high-load HTTP service that needed to perform parallel database queries. The first thing I learned was that **context** must be passed from the handler to the goroutine at every step, because without it, we couldn’t cancel all goroutines when the client closed the connection or a timeout occurred. I used `context.WithCancel` in the middleware to create a cancel function, and then each worker checked the `Done()` channel of the context before starting any heavy I/O operations. For the **worker pool**, I set a fixed number of workers (e.g., `runtime.NumCPU()*2`), created a `jobs` channel for incoming tasks, and had each worker consume from it and send results to a `results` channel. The key part here was closing the channels in an orderly way: first close `jobs` after all tasks were sent, then use `sync.WaitGroup` to wait for all workers to finish before closing `results`. This pattern prevented goroutine leaks that could run indefinitely. As for **error handling**, my preferred pattern is returning errors from every function and wrapping them with `fmt.Errorf("operation X failed: %w", err)` to preserve the stack trace. Inside the workers, if a panic occurred, I used a `defer` with `recover()`, logged the error, and sent a failure signal through an error channel to be tracked by the main routine. I also used `errgroup.Group` from the `x/sync` package to integrate context and error management seamlessly—if any goroutine returned an error or panicked, the group would automatically close the context and stop the remaining workers. This way, I could manage cancellation, errors, and resource cleanup without falling into common traps like goroutine leaks or deadlocks.