Collecting and analysing data after an earthquake is essential to determine its impact. In 2014, the European Mediterranean Seismological Centre (EMSC) launched the LastQuake system. This system collects intensity reports from users to help provide rapid situational awareness. However, text data collected through crowdsourcing platforms is unstructured. Therefore, natural language processing techniques such as sentiment analysis are necessary to extract meaningful information. On the 26th November 2019, following an earthquake in Albania, the LastQuake app recorded 28,220 reports with user comments. For the current analysis, we sampled comments posted on the exact day of the earthquake, in Albanian: 1678 comments (6%). The most frequent polarity detected in comments from LastQuake app users was negative (52%) followed by far by positive, neutral and unrelated comments. However, manual classification is time-consuming and not feasible during the emergency phase. Therefore, we tested the accuracy of two automatic classification models for sentiment analysis: ‘troberta’ and ‘txlm’. These models were fine-tuned using already classified text data from the 2020 Aegean earthquake. Using the manual classification as the reference to evaluate the accuracy of automatic classification models for sentiment analysis yields accuracies of 71% for the ‘troberta’ model and 56% for the ‘txlm’ model.