Evaluating learning rate effects on long short-term memory for Indonesian sentiment classification

Abstract

Hyperparameter optimization is a crucial process for enhancing the performance of deep learning models, particularly in the context of Indonesian sentiment classification. This study examines the impact of varying learning rates on a long short-term memory (LSTM) architecture trained with the adaptive moment estimation (Adam) optimizer. The dataset comprises 9,295 Indonesian comments automatically labeled by the Indonesian Bidirectional Encoder Representations from Transformers (IndoBERT) model. Stratified k-fold cross-validation was employed to maintain class balance during training. Learning curves were analyzed to evaluate convergence and identify potential overfitting, while early stopping was applied when performance improvements became insignificant. The one-way analysis of variance (ANOVA) test (p-adj = 0.000575 < 0.05) revealed significant differences among the learning rate variations. Post-hoc analysis indicated the learning rates of 0.0001, 0.001, and 0.002 differ significantly from 0.02. Descriptive statistics showed that a learning rate of 0.001 was the most optimal, achieving the highest validation accuracy while maintaining a relatively low variance. Evaluation across two data categories demonstrated that lower learning rates (0.0001 and 0.002) achieved the best accuracy, 78.71% on in-domain data, whereas higher learning rates (0.01 and 0.02) performed better on cross-domain data with 36% accuracy. These findings highlight the crucial role of learning rate selection in determining model stability and generalization capability.