The rapid growth of digital banking and online financial services has significantly increased the risk and complexity of financial fraud, demanding intelligent and scalable detection mechanisms. Traditional rule-based systems are often inadequate due to high false-positive rates and limited adaptability to evolving fraud patterns. To address these challenges, this study proposes an AI- and Big Data–driven fraud detection framework that integrates machine learning and deep learning techniques for accurate and real-time fraud identification. The proposed methodology employs XGBoost and Long Short-Term Memory (LSTM) models, along with a novel hybrid LSTM–XGBoost architecture, to capture both transactional patterns and temporal behavioral characteristics from large-scale banking transaction data. Extensive experiments conducted on a real-world benchmark dataset demonstrate the effectiveness of the proposed approach. The hybrid model achieves superior performance with an accuracy of 0.989, precision of 0.907, recall of 0.946, F1-score of 0.926, and AUC of 0.987, while also significantly reducing the false positive rate to 0.021. Furthermore, scalability analysis confirms its suitability for big data environments with efficient training and low inference latency. Overall, the results indicate that the proposed framework offers a robust, accurate, and scalable solution for fraud detection in modern banking systems.