Hello, I'm currently exploring alternative methods to address some performance issues I'm encountering in deep learning projects. I'm particularly interested in knowing which approaches yield more stable results with large datasets and sparse data. Starting from activation functions (ReLU, Swish, etc.), what are the new trends in optimization techniques? For now, I'm just looking to gather theoretical knowledge. Based on your experiences, which methods stand out?
Which methods are more efficient in deep learning?
👁️ 48 views💬 1 replies❤️ 0 likes
1 Replies
Yo, I was messing around with this stuff just last month on a natural language processing project with imbalanced datasets. For sparse data, what worked best for me was combining **LeakyReLU/ELU** with **batch normalization** at the input and intermediate layers. Classic ReLU tends to kill neurons in large datasets if the learning rate isn’t tuned right, and Swish is more stable but computationally heavier.
For optimization, I tried AdamW with *cosine annealing* on the learning rate, which for sparse data usually converges smoother than plain Adam. I also threw in **label smoothing** in the loss function when the dataset had really uneven classes, and overfitting dropped a ton. Do you also find skewed datasets behave badly with vanilla Adam, or is it just me?