Publications

2026

1. A Moodle Plugin for Handwritten Answer Input, Management, and Recognition Supporting Japanese, English, Mathematical, and Chemical Formulas

   Thanh-Nghia Truong, Hung Tuan Nguyen, Fumiko Yasuno, and 4 more authors

   IEICE Transactions on Information and Systems, 2026
2. Link Prediction Graph Neural Networks for Structure Recognition of Handwritten Mathematical Expressions

   Cuong Tuan Nguyen, Ngoc Tuan Nguyen, Triet Hoang Minh Dao, and 2 more authors

   In Document Analysis and Recognition – ICDAR 2025 Workshops, 2026
3. Integrating Visual Attention into Deep Reinforcement Learning for Enhanced Control in Racing Games

   Khanh-Linh Vuong, Huy Truong Dinh, and Cuong Tuan Nguyen

   In Computational Intelligence in Engineering Science, 2026

   Series Title: Communications in Computer and Information Science
4. Forecasting Weather: A Scenario of the Mekong Delta

   Kim-Thanh Hoang Le, Cam-Tu Ha, Cuong Tuan Nguyen, and 1 more author

   In Computational Intelligence in Engineering Science, 2026

   Series Title: Communications in Computer and Information Science

2025

1. A MINLP Framework for Day-Ahead Scheduling of Fast Electric Vehicle Charging Stations Integrated Wind and Solar Energy Considering Load Balancing and Energy Trading

   Huy Truong Dinh, Truong Hoang Bao Huy, Nguyen Dang The Vinh, and 1 more author

   IEEE Access, 2025
2. SwinTExCo: Exemplar-based video colorization using Swin Transformer

   Duong Thanh Tran, Nguyen Doan Hieu Nguyen, Trung Thanh Pham, and 5 more authors

   Expert Systems with Applications, Jan 2025
3. Synthetic Swarm Mosquito Dataset for Acoustic Classification: A Proof of Concept

   Thai Duy Dinh, Minh Luan Vo, Cuong Tuan Nguyen, and 1 more author

   In 2025 RIVF International Conference on Computing and Communication Technologies (RIVF), Dec 2025
4. A Real-Time Vehicle Detection Pipeline with Data-Centric Enhancements and Multi-Stage DETR Distillation

   Huy Minh Nhat Nguyen, Hieu Dinh Trung Pham, Khang Minh Le, and 1 more author

   In 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Oct 2025
5. SDPA++: A General Framework for Self-Supervised Denoising with Patch Aggregation

   Huy Minh Nhat Nguyen, Triet Hoang Minh Dao, Chau Vinh Hoang Truong, and 1 more author

   In 2025 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), Aug 2025

2024

1. A survey on handwritten mathematical expression recognition: The rise of encoder-decoder and GNN models

   Thanh-Nghia Truong, Cuong Tuan Nguyen, Richard Zanibbi, and 2 more authors

   Pattern Recognition, Sep 2024
2. Performance Comparison in Traffic Sign Recognition Using Deep Learning

   Binh Dang Hai, Huu Duc Nguyen, Thanh Nam Vo, and 3 more authors

   In Industrial Networks and Intelligent Systems, Sep 2024

   Series Title: Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
3. MERSA: Multimodal Emotion Recognition with Self-Align Embedding

   Quan Bao Le, Kiet Tuan Trinh, Nguyen Dinh Hung Son, and 3 more authors

   In 2024 International Conference on Information Networking (ICOIN), Jan 2024
4. Towards Real-Time Vietnamese Traffic Sign Recognition on Embedded Systems

   Phuong-Nam Tran, Nhat Truong Pham, Nam Van Hai Phan, and 3 more authors

   In 2024 15th International Conference on Information and Communication Technology Convergence (ICTC), Oct 2024
5. Two Experiments for Automatic Scoring of Handwritten Descriptive Answers

   Masaki Nakagawa, Hung Tuan Nguyen, Nghia Thanh Truong, and 9 more authors

   In Document Analysis Systems, Oct 2024

   Series Title: Lecture Notes in Computer Science

2023

1. ICDAR 2023 CROHME: Competition on Recognition of Handwritten Mathematical Expressions

   Yejing Xie, Harold Mouchère, Foteini Simistira Liwicki, and 5 more authors

   In Document Analysis and Recognition - ICDAR 2023, Oct 2023

   Series Title: Lecture Notes in Computer Science
2. Incremental Teacher Model with Mixed Augmentations and Scheduled Pseudo-label Loss for Handwritten Text Recognition

   Masayuki Honda, Hung Tuan Nguyen, Cuong Tuan Nguyen, and 4 more authors

   In Document Analysis and Recognition - ICDAR 2023, Oct 2023

   Series Title: Lecture Notes in Computer Science
3. RBBA: ResNet - BERT - Bahdanau Attention for Image Caption Generator

   Duc-Hieu Hoang, Anh-Khoa Tran, Duc Ngoc Minh Dang, and 3 more authors

   In 2023 14th International Conference on Information and Communication Technology Convergence (ICTC), Oct 2023
4. Digitalizing educational workbooks and collecting handwritten answers for automatic scoring

   Tomo Asakura, Hung Nguyen, Nghia Truong, and 12 more authors

   In Proceedings of the Fifth International Workshop on Intelligent Textbooks 2023 co-located with the 24th International Conference on Artificial Intelligence in Education {(AIED} 2023), Tokyo, Japan, July 3, 2023, Oct 2023

   Series Title: {CEUR} Workshop Proceedings

2022

1. Syntactic data generation for handwritten mathematical expression recognition

   Thanh Nghia Truong, Cuong Tuan Nguyen, and Masaki Nakagawa

   Pattern Recognition Letters, Oct 2022

   Abs

   This paper proposes tree-based decomposition and sub-expression interchange for generating new syntactically valid handwritten mathematical expressions (HMEs) from a given set of HMEs to train an HME recognition model and a mathematical language model (LM). The recognition model is dual trained using weakly supervised learning and encoder-decoder attention loss on the generated samples. Recognition experiments indicate that the proposed data generation method is superior to other such methods for offline HMEs. The HME recognition model increases the expression recognition rates by 1.47, 2.88, and 2.67 points on the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014, 2016, and 2019 testing sets, respectively. The LM also increases them by 8.92, 6.88, and 2.59 points on the testing sets. Further adding extra LaTeX sequences is cost effective in strengthening the LM for the expression recognition rates being 2.54, 2.8, and 1.25 points higher than without them on the CROHME testing sets, respectively. Among academic systems, the trained HME recognition system achieves the best performance with 64.60% and 66.08% expression recognition rates on the CROHME 2014 and 2016 testing sets and a comparable expression recognition rate of 58.72% on the CROHME 2019 testing set, respectively. Comparison with top systems from companies on CROHME 2019 suggests that more real and/or generated HME patterns will improve the performance of HME recognition models as well as mathematical language models.

2. Handwriting Recognition and Automatic Scoring for Descriptive Answers in Japanese Language Tests

   Hung Tuan Nguyen, Cuong Tuan Nguyen, Haruki Oka, and 2 more authors

   In Frontiers in Handwriting Recognition, Oct 2022

   Series Title: Lecture Notes in Computer Science
3. Learning Symbol Relation Tree for Online Handwritten Mathematical Expression Recognition

   Thanh Nghia Truong, Hung Tuan Nguyen, Cuong Tuan Nguyen, and 1 more author

   Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Oct 2022

   ISBN: 978-3-031-02444-3

   Abs

   This paper proposes a method for recognizing online handwritten mathematical expressions (OnHME) by building a symbol relation tree (SRT) directly from a sequence of strokes. The recognition system has two parts: a temporal classifier and a tree connector. The...

4. Fully Automated Short Answer Scoring of the Trial Tests for Common Entrance Examinations for Japanese University

   Haruki Oka, Hung Tuan Nguyen, Cuong Tuan Nguyen, and 2 more authors

   In Proceedings - 23rd International Conference Artificial Intelligence in Education, AIED 2022, Oct 2022

   Abs

   Studies on automated short-answer scoring (SAS) have been conducted to apply natural language processing to education. Short-answer scoring is a task to grade the responses from linguistic information. Most answer sheets for short-answer questions are handwritten in an actual educational setting, which is a barrier to SAS. Therefore, we have developed a system that uses handwritten character recognition and natural language processing for fully automated scoring of handwritten responses to short-answer questions. This is the most extensive scoring data for responses to short-answer questions, and it may be the largest in the world. Applying the Cohen’s kappa coefficient to the graded evaluations, the results show 0.86 in the worst case, and approximately 0.95 is recorded for the remaining five question answers. We observe that the fully automated scoring system proposed in our study can also score with a high degree of accuracy comparable to that of human scoring.

2021

1. Clustering of handwritten mathematical expressions for computer-assisted marking

   Vu Tran Minh Khuong, Khanh Minh Phan, Huy Quang Ung, and 2 more authors

   IEICE Transactions on Information and Systems, Oct 2021

   Abs

   Many approaches enable teachers to digitalize students’ answers and mark them on the computer. However, they are still limited for supporting marking descriptive mathematical answers that can best evaluate learners’ understanding. This paper presents clustering of offline handwritten mathematical expressions (HMEs) to help teachers efficiently mark answers in the form of HMEs. In this work, we investigate a method of combining feature types from low-level directional features and multiple levels of recognition: Bag-of-symbols, bag-of-relations, and bag-ofpositions. Moreover, we propose a marking cost function to measure the marking effort. To show the effectiveness of our method, we used two datasets and another sampled from CROHME 2016 with synthesized patterns to prepare correct answers and incorrect answers for each question. In experiments, we employed the k-means++ algorithm for each level of features and considered their combination to produce better performance. The experiments show that the best combination of all the feature types can reduce the marking cost to about 0.6 by setting the number of answer clusters appropriately compared with the manual one-by-one marking.

2. Clustering online handwritten mathematical expressions

   Huy Quang Ung, Cuong Tuan Nguyen, Khanh Minh Phan, and 2 more authors

   Pattern Recognition Letters, Jun 2021

   Abs

   To help human markers mark many answers in the form of online handwritten mathematical expressions (OHMEs), this paper proposes bag-of-features for clustering OHMEs. It consists of six levels of features from low-level pattern features to high-level symbolic and structural features obtained from a state-of-the-art OHME recognizer. Then, it introduces distance-based representation (DbR) to reduce the dimensionality of our proposed feature spaces. Moreover, it presents a method for combining the proposed features to improve the performance. Experiments using the k-means++ algorithm are conducted on a set of 3,150 OHMEs (Dset_50) and an answer dataset (Dset_Mix) of 200 OHMEs intermixed between real patterns and synthesized patterns for each of 10 questions. When the number of clusters is set as the true number of categories, the best purity around 0.99 is produced by bag-of-symbols with DbR for Dset_50, which is better than state-of-the-art methods for clustering offline patterns converted from their OHMEs. The combination of both low-level and high-level features with DbR achieves a purity of around 0.777, increases to more than 0.90 and reduce the marking cost by more than 0.35 point than manually marking OHME answers by adjusting the number of clusters for Dset_Mix.

3. Global Context for Improving Recognition of Online Handwritten Mathematical Expressions

   Cuong Tuan Nguyen, Thanh Nghia Truong, Hung Tuan Nguyen, and 1 more author

   Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Jun 2021

   ISBN: 9783030863302

   Abs

   This paper presents a temporal classification method for all three subtasks of symbol segmentation, symbol recognition and relation classification in online handwritten mathematical expressions (HMEs). The classification model is trained by multiple paths of symbols and spatial relations derived from the Symbol Relation Tree (SRT) representation of HMEs. The method benefits from global context of a deep bidirectional Long Short-term Memory network, which learns the temporal classification directly from online handwriting by the Connectionist Temporal Classification loss. To recognize an online HME, a symbol-level parse tree with Context-Free Grammar is constructed, where symbols and spatial relations are obtained from the temporal classification results. We show the effectiveness of the proposed method on the two latest CROHME datasets.

4. A Transformer-Based Math Language Model for Handwritten Math Expression Recognition

   Huy Quang Ung, Cuong Tuan Nguyen, Hung Tuan Nguyen, and 2 more authors

   In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR 2021 Workshops, Jun 2021

   Abs

   Handwritten mathematical expressions (HMEs) contain ambiguities in their interpretations, even for humans sometimes. Several math symbols are very similar in the writing style, such as dot and comma or “0”, “O”, and “o”, which is a challenge for HME recognition systems to handle without using contextual information. To address this problem, this paper presents a Transformer-based Math Language Model (TMLM). Based on the self-attention mechanism, the high-level representation of an input token in a sequence of tokens is computed by how it is related to the previous tokens. Thus, TMLM can capture long dependencies and correlations among symbols and relations in a mathematical expression (ME). We trained the proposed language model using a corpus of approximately 70,000 LaTeX sequences provided in CROHME 2016. TMLM achieved the perplexity of 4.42, which outperformed the previous math language models, i.e., the N-gram and recurrent neural network-based language models. In addition, we combine TMLM into a stochastic context-free grammar-based HME recognition system using a weighting parameter to re-rank the top-10 best candidates. The expression rates on the testing sets of CROHME 2016 and CROHME 2019 were improved by 2.97 and 0.83 percentage points, respectively.

5. Temporal Classification Constraint for Improving Handwritten Mathematical Expression Recognition

   Cuong Tuan Nguyen, Hung Tuan Nguyen, Kei Morizumi, and 1 more author

   In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR 2021 Workshops, Jun 2021

   Abs

   We present a temporal classification constraint as an auxiliary learning method for improving the recognition of Handwritten Mathematical Expression (HME). Connectionist temporal classification (CTC) is used to learn the temporal alignment of the input feature sequence and corresponding symbol label sequence. The CTC alignment is trained with the encoder-decoder alignment through a combination of CTC loss and encoder-decoder loss to improve the feature learning of the encoder in the encoder-decoder model. We show the effectiveness of the approach in improving symbol classification and expression recognition on the CROHME datasets.

6. Relation-Based Representation for Handwritten Mathematical Expression Recognition

   Thanh Nghia Truong, Huy Quang Ung, Hung Tuan Nguyen, and 2 more authors

   In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR 2021 Workshops, Jun 2021

   Abs

   This paper proposes relation-based sequence representation that enhances offline handwritten mathematical expressions (HMEs) recognition. Commonly, a LaTeX-based sequence represents the 2D structure of an HME as a 1D sequence. Consequently, the LaTeX-based sequence becomes longer, and HME recognition systems have difficulty in extracting its 2D structure. We propose a new representation for HMEs according to the relations of symbols, which shortens the LaTeX-based representation. We use an offline end-to-end HME recognition system that adopts weakly supervised learning to evaluate the proposed representation. Recognition experiments indicate that the proposed relation-based representation helps the HME recognition system achieve higher performance than the LaTeX-based representation. In fact, the HME recognition system achieves recognition rates of 53.35%, 52.14%, and 53.13% on the dataset of the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014, 2016, and 2019, respectively. These results are more than 2 percentage points higher than the LaTeX-based system.

7. GSSF: A Generative Sequence Similarity Function Based on a Seq2Seq Model for Clustering Online Handwritten Mathematical Answers

   Huy Quang Ung, Cuong Tuan Nguyen, Hung Tuan Nguyen, and 1 more author

   In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR 2021, Jun 2021

   Abs

   Toward a computer-assisted marking for descriptive math questions, this paper presents clustering of online handwritten mathematical expressions (OnHMEs) to help human markers to mark them efficiently and reliably. We propose a generative sequence similarity function for computing a similarity score of two OnHMEs based on a sequence-to-sequence OnHME recognizer. Each OnHME is represented by a similarity-based representation (SbR) vector. The SbR matrix is inputted to the k-means algorithm for clustering OnHMEs. Experiments are conducted on an answer dataset (Dset_Mix) of 200 OnHMEs mixed of real patterns and synthesized patterns for each of 10 questions and a real online handwritten mathematical answer dataset of 122 student answers at most for each of 15 questions (NIER_CBT). The best clustering results achieved around 0.916 and 0.915 for purity, and around 0.556 and 0.702 for the marking cost on Dset_Mix and NIER_CBT, respectively. Our method currently outperforms the previous methods for clustering HMEs.

2020

1. Nom document digitalization by deep convolution neural networks

   Kha Cong Nguyen, Cuong Tuan Nguyen, and Masaki Nakagawa

   Pattern Recognition Letters, Jun 2020

   Abs

   Nom is an ancient script used in Vietnam until the current Latin-based Vietnamese alphabet became common, and a large number of ancient Nom documents are in existence. Due to the gradual degradation of Nom documents and a decrease in the number of scholars who can understand them, a system to digitalize Nom documents is urgently necessary. This paper presents a segmentation-based method for digitalizing Nom documents using deep convolution neural networks. Nom pages are preprocessed, segmented into isolated characters, and then recognized by a single-character OCR. The structure of the U-Net is applied to create segmentation maps and extract character regions from them. Subsequently, we propose coarse and fine combined classifiers to recognize each character pattern. The results by the best classifier are revised by a decoder using a langue model. The decoder is the same as the connectionist temporal classification decoder used in end-to-end text recognition systems. Compared with the traditional segmentation method using projection profiles and the Voronoi diagram (IoU = 81.23%), the segmentation method using the deep convolution neural network produces a better result (IoU = 92.08%) for detecting character regions. The proposed CNN models for recognizing segmented character patterns outperforms the traditional models using the modified quadratic discriminant function and the learning vector quantization with the recognition rate of 85.07%. The combination of coarse and fine classifiers, the training dataset with salt and pepper noises, and the attention layer are the key factors in the recognition rate improvement.

2. An attention-based row-column encoder-decoder model for text recognition in Japanese historical documents

   Nam Tuan Ly, Cuong Tuan Nguyen, and Masaki Nakagawa

   Pattern Recognition Letters, Jun 2020

   Abs

   This paper presents an attention-based row-column encoder-decoder (ARCED) model for recognizing an input image of multiple text lines from Japanese historical documents without explicit segmentation of lines. The recognition system has three main parts: a feature extractor, a row-column encoder, and a decoder. We introduce a row-column BLSTM in the encoder and a residual LSTM network in the decoder. The whole system is trained end-to-end by a standard cross-entropy loss function, requiring only document images and their ground-truth text. We experimentally evaluate the performance of ARCED on the dataset of Japanese historical documents: Kana-PRMU. The results of the experiments show that ARCED outperforms the state-of-the-art recognition methods on the dataset. Furthermore, we demonstrate that the row-column BLSTM in the encoder and the residual LSTM in the decoder improves the performance of the encoder-decoder model for the recognition of Japanese historical document.

3. CNN based spatial classification features for clustering offline handwritten mathematical expressions

   Cuong Tuan Nguyen, Vu Tran Minh Khuong, Hung Tuan Nguyen, and 1 more author

   Pattern Recognition Letters, Jun 2020

   Abs

   To help human markers mark a large number of answers of handwritten mathematical expressions (HMEs), clustering them makes marking more efficient and reliable. Clustering HMEs, however, faces the problem of extracting both localization and classification representation of mathematical symbols for an HME image and defining the distance between two HME images. First, we propose a method based on Convolutional Neural Networks (CNN) to extract the representations for an HME. Symbols in various scales are located and classified by a combination of features from a multi-scale CNN. We use weakly supervised training combined with symbols attention to enhance localization and classification predictions. Second, we propose a multi-level spatial distance between two representations for clustering HMEs. Experiments on CROHME 2016 and CROHME 2019 dataset show the promising results of 0.99 and 0.96 in purity, respectively.

4. A unified method for augmented incremental recognition of online handwritten Japanese and English text

   Cuong Tuan Nguyen, Bipin Indurkhya, and Masaki Nakagawa

   International Journal on Document Analysis and Recognition, Jun 2020

   Abs

   We present a unified method to augmented incremental recognition for online handwritten Japanese and English text, which is used for busy or on-the-fly recognition while writing, and lazy or delayed recognition after writing, without incurring long waiting times. It extends the local context for segmentation and recognition to a range of recent strokes called “segmentation scope” and “recognition scope,” respectively. The recognition scope is inside of the segmentation scope. The augmented incremental recognition triggers recognition at every several recent strokes, updates the segmentation and recognition candidate lattice, and searches over the lattice for the best result incrementally. It also incorporates three techniques. The first is to reuse the segmentation and recognition candidate lattice in the previous recognition scope for the current recognition scope. The second is to fix undecided segmentation points if they are stable between character/word patterns. The third is to skip recognition of partial candidate character/word patterns. The augmented incremental method includes the case of triggering recognition at every new stroke with the above-mentioned techniques. Experiments conducted on TUAT-Kondate and IAM online database show its superiority to batch recognition (recognizing text at one time) and pure incremental recognition (recognizing text at every input stroke) in processing time, waiting time, and recognition accuracy.

5. A Semantic Segmentation-based Method for Handwritten Japanese Text Recognition

   Kha Cong Nguyen, Cuong Tuan Nguyen, and Masaki Nakagawa

   In Proceedings of International Conference on Frontiers in Handwriting Recognition, ICFHR 2020, Jun 2020

   Abs

   Recently, the segmentation-free approach using Convolution Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) for handwritten text recognition has been investigated by many research groups. Although good results are produced on some datasets, there remain some drawbacks. It is neither robust against the change of handwriting styles such as gaps between characters, stroke widths and shapes of characters nor stable for skewed, and curved text lines unless heavily trained by such patterns. In this paper, we propose a segmentation-based method for handwritten Japanese text recognition. The method employs a semantic segmentation model for precisely splitting text lines into single characters. The semantic segmentation model is based on an encoder-decoder architecture like U-Net, but we employ available techniques to improve the accuracy of pixel classification. They are a deeper encoder with ResNet 101, dilated convolutions and Spatial Pyramid Pooling. Subsequently, a CNN based OCR is used to recognize segmented character images. Finally, the recognized candidates are considered in a lattice diagram combined with a linguistic context. The result of experiments shows that the segmentation model increases the mean IoU significantly from 89.32% to 94.96% and the proposed approach is robust to the change of handwriting styles while a segmentation-free method is quite sensitive to them, resulting in the significant reduction of the recognition rate.

6. Online trajectory recovery from offline handwritten Japanese kanji characters of multiple strokes

   Hung Tuan Nguyen, Tsubasa Nakamura, Cuong Tuan Nguyen, and 1 more author

   In Proceedings - International Conference on Pattern Recognition, ICPR 2020, Jun 2020

   Abs

   We propose a deep neural network-based method to recover dynamic online trajectories from offline handwritten Japanese kanji character images. It is a challenging task since Japanese kanji characters consist of multiple strokes. Our proposed model has three main components: Convolutional Neural Network-based encoder, Long Short-Term Memory Network-based decoder with an attention layer, and Gaussian Mixture Model (GMM). The encoder focuses on feature extraction while the decoder refers to the extracted features and generates time-sequences of GMM parameters. The attention layer is the key component for trajectory recovery. The GMM provides robustness to style variations so that the proposed model does not overfit to training samples. In the experiments, the proposed method is evaluated by both visual verification and handwritten character recognition. This is the first attempt to use online recovered trajectories to help improve offline handwriting recognition performance. Although the visual verification reveals some problems, the recognition experiments demonstrate the effect of trajectory recovery in improving offline handwritten character recognition accuracy when online recognition of the recovered trajectories are combined.

7. Attention Augmented Convolutional Recurrent Network for Handwritten Japanese Text Recognition

   Nam Tuan Ly, Cuong Tuan Nguyen, and Masaki Nakagawa

   In Proceedings of International Conference on Frontiers in Handwriting Recognition, ICFHR 2020, Jun 2020

   Abs

   Handwritten Japanese text recognition is still a big challenging task due to the large character set, diversity of writing styles, and multiple-touches between characters. In this paper, we propose a model of Attention Augmented Convolutional Recurrent Network (AACRN) for recognizing handwritten Japanese text lines. The AACRN model has three main parts: A convolutional feature extractor, a self-attention based encoder, and a CTC-decoder. The whole model can be trained end-to-end. In the experiment, we evaluate the performance of the AACRN model on the TUAT Kondate dataset and the Kuzushiji dataset. The results of the experiments show that the proposed model achieves higher performance than the state-of-the-art recognition accuracies on the test set of TUAT Kondate and the Kuzushiji dataset.

8. A Siamese Network-based Approach for Matching Various Sizes of Excavated Wooden Fragments

   Trung Tan Ngo, Cuong Tuan Nguyen, and Masaki Nakagawa

   In Proceedings of International Conference on Frontiers in Handwriting Recognition, ICFHR 2020, Jun 2020

   Abs

   This paper presents an approach for matching various sizes of excavated wooden fragments based on siamese neural networks. We propose a siamese network composed of global average pooling and a fine-tuned Resnet encoder (GA-S-net). We also propose an elaborated siamese network by replacing the global average pooling with a spatial pyramid pooling layer and add a new dense absolute difference layer (SP-S-net). Samples of 37,760 fragments were prepared from 268 complete wooden tablets excavated from the Heijo-Kyo Palace ruins used during the Nara period in Japan. Both of the networks answer whether two fragments are from the same tablet or not. The result of both networks for the testing set is similar to AUC (Area under the curve) of ROC (Receiver Operating Characteristic) curve being around 90%. In AUC of large fragments, however, SP-S-net is better than GA-S-net with 97.1% versus 93.8%. These networks are rather new for dealing with various sizes of inputs for the matching problem.

9. Online Handwritten Mathematical Symbol Segmentation and Recognition with Bidirectional Context

   Cuong Tuan Nguyen, Thanh Nghia Truong, Huy Quang Ung, and 1 more author

   In Proceedings of International Conference on Frontiers in Handwriting Recognition, ICFHR, Sep 2020

   Abs

   Discriminating ambiguous symbols in online handwritten mathematical expression is difficult without context. We propose a Bidirectional Recurrent Neural Network for segmenting and classifying online handwritten mathematical symbols. The context from forward and backward directions helps the classification model discriminate ambiguous symbols and improve recognition rates. The classification model is integrated into the Stochastic Context-Free Grammar recognition system for recognizing mathematical expressions. We show the effectiveness of the approach for improving symbol classification and segmentation on the CROHME 2016 dataset.

10. Improvement of End-to-End Offline Handwritten Mathematical Expression Recognition by Weakly Supervised Learning

    Thanh Nghia Truong, Cuong Tuan Nguyen, Khanh Minh Phan, and 1 more author

    In Proceedings of International Conference on Frontiers in Handwriting Recognition, ICFHR, Sep 2020

    Abs

    This paper presents an improvement in recognizing offline handwritten mathematical expressions (HMEs) by deep neural networks. We train it end-to-end using weakly supervised learning. The network has three parts: An encoder using a Convolutional Neural Network to encode high-level features from an input HME image; a decoder using gated recurrent units with attention to parse the high-level features and generate an output expression in the LaTeX format; and a symbol classifier to improve the localization and classification of the high-level features. Besides, we use the model ensemble method for the beam search process to average the probabilities from multiple models. For the dataset of the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014 and 2016, we have achieved an expression recognition rate of 53.65% and 51.96% correspondingly, which is 6 points better than without weakly supervised learning. Furthermore, when ensembling several models, the recognition rate of our method is increased to 55.68% for the CROHME 2014 testing set.

2019

1. Recognition of anomalously deformed kana sequences in Japanese historical documents

   Nam Tuan Ly, Kha Cong Nguyen, Cuong Tuan Nguyen, and 1 more author

   IEICE Transactions on Information and Systems, Sep 2019

   Abs

   This paper presents recognition of anomalously deformed Kana sequences in Japanese historical documents, for which a contest was held by IEICE PRMU 2017. The contest was divided into three levels in accordance with the number of characters to be recognized: Level 1: Single characters, level 2: Sequences of three vertically written Kana characters, and level 3: Unrestricted sets of characters composed of three or more characters possibly in multiple lines. This paper focuses on the methods for levels 2 and 3 that won the contest. We basically follow the segmentationfree approach and employ the hierarchy of a Convolutional Neural Network (CNN) for feature extraction, Bidirectional Long Short-Term Memory (BLSTM) for frame prediction, and Connectionist Temporal Classification (CTC) for text recognition, which is named a Deep Convolutional Recurrent Network (DCRN). We compare the pretrained CNN approach and the end-to-end approach with more detailed variations for level 2. Then, we propose a method of vertical text line segmentation and multiple line concatenation before applying DCRN for level 3. We also examine a twodimensional BLSTM (2DBLSTM) based method for level 3. We present the evaluation of the best methods by cross validation. We achieved an accuracy of 89.10% for the three-Kana-character sequence recognition and an accuracy of 87.70% for the unrestricted Kana recognition without employing linguistic context. These results prove the performances of the proposed models on the level 2 and 3 tasks.

2. Text-independent writer identification using convolutional neural network

   Hung Tuan Nguyen, Cuong Tuan Nguyen, Takeya Ino, and 2 more authors

   Pattern Recognition Letters, Apr 2019

   Abs

   The text-independent approach to writer identification does not require the writer to write some predetermined text. Previous research on text-independent writer identification has been based on identifying writer-specific features designed by experts. However, in the last decade, deep learning methods have been successfully applied to learn features from data automatically. We propose here an end-to-end deep-learning method for text-independent writer identification that does not require prior identification of features. A Convolutional Neural Network (CNN) is trained initially to extract local features, which represent characteristics of individual handwriting in the whole character images and their sub-regions. Randomly sampled tuples of images from the training set are used to train the CNN and aggregate the extracted local features of images from the tuples to form global features. For every training epoch, the process of randomly sampling tuples is repeated, which is equivalent to a large number of training patterns being prepared for training the CNN for text-independent writer identification. We conducted experiments on the JEITA-HP database of offline handwritten Japanese character patterns. With 200 characters, our method achieved an accuracy of 99.97% to classify 100 writers. Even when using 50 characters for 100 writers or 100 characters for 400 writers, our method achieved accuracy levels of 92.80% or 93.82%, respectively. We conducted further experiments on the Firemaker and IAM databases of offline handwritten English text. Using only one page per writer to train, our method achieved over 91.81% accuracy to classify 900 writers. Overall, we achieved a better performance than the previously published best result based on handcrafted features and clustering algorithms, which demonstrates the effectiveness of our method for handwritten English text also.

3. Robust and real-time stroke order evaluation using incremental stroke context for learners to write Kanji characters correctly

   Cuong Tuan Nguyen, Hung Tuan Nguyen, Kazuhiro Mita, and 1 more author

   Pattern Recognition Letters, Apr 2019

   Abs

   Writing Kanji characters of Chinese origin in the correct stroke order and direction is still one of the important subjects in Japanese elementary education. So far, the stroke order evaluation was made by stroke-to-stroke matching without stroke context so that it was unrobust to Kanji characters having multiple similar strokes. Here, we employ shape context features around each feature point in not only conventional fan-shaped bins but also in square bins with applying a Gaussian function. We also propose simple incremental context and augmented context from future strokes. Our approach can judge whether the stroke order and direction are correct or not every time a new stroke is written on a tablet by matching a partially written Kanji pattern with the reference pattern written to the same number of strokes. Evaluation shows that the best-tuned method with square bins and the Gaussian function records the highest performance and correctly evaluates stroke order by 98.5% with the maximum time of 0.12 sec. /character for Kanji patterns after all strokes are written using an average desktop PC. The method is also shown to possess high reliability to detect wrong stroke order and direction incrementally every time after each stroke is written.

4. An online overlaid handwritten Japanese text recognition system for small tablet

   Jianjuan Liang, Cuong Tuan Nguyen, Bilan Zhu, and 1 more author

   Pattern Analysis and Applications, Apr 2019

   Abs

   The paper presents a recognition system of online overlaid handwritten Japanese text patterns on a smart phone or baby-face tablet. The proposed system oversegments a sequence of strokes into primitive segments at candidate off-strokes between strokes using a SVM model. One or more consecutive primitive segments form a candidate character pattern, which is recognized into a list of candidate categories. Then, a segmentation and recognition candidate lattice is constructed to represent all candidate character patterns and their corresponding character classes. Finally, the optimal path is effectively found by the Viterbi search in the lattice, combining the scores of character recognition, geometric features, linguistic context, as well as the segmentation scores by SVM classification. This system incorporates feature reduction and non-character pruning to decrease the time cost per character, and semi-incremental recognition to decrease waiting time. The recognition rates on generated and collected overlaid handwritten text are 92.16% and 93.04%, respectively. The average time cost per character is not more than 0.6 s, and the average waiting time is less than 0.875 s even on an Intel Atom 1.33 GHz CPU: a low power consumption CPU for small tablets and embedded devices. Therefore, we confirm that our system recognizes online overlaid handwritten text composed of thousands of Japanese character classes with the high recognition rate without excessive waiting time.

5. A Character Attention Generative Adversarial Network for Degraded Historical Document Restoration

   Kha Cong Nguyen, Cuong Tuan Nguyen, Seiji Hotta, and 1 more author

   In 2019 International Conference on Document Analysis and Recognition (ICDAR), Sep 2019
6. Deep Convolutional Recurrent Network for Segmentation-Free Offline Handwritten Japanese Text Recognition

   Nam Tuan Ly, Cuong Tuan Nguyen, Kha Cong Nguyen, and 1 more author

   In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR 2019, Sep 2019

   Abs

   This paper presents a model of Deep Convolutional Recurrent Network (DCRN) for recognizing offline handwritten Japanese text lines without explicit segmentation of characters. Most of traditional offline handwritten Japanese text recognizers perform segmentation of text image into characters before individually recognizing each character. Although segmentation by recognition and context are employed to recover from segmentation errors, errors made at this stage directly make an impact on the performance of the whole system. The DCRN model consists of three parts: a convolutional feature extractor using Convolutional Neural Network (CNN) and sliding window to extract features from text image; recurrent layers using BLSTM to predict pre-frame from an input sequence; and a transcription layer using a CTC-decoder to translate the predictions into the label sequence. Experimental results on the database: TUAT Kondate database demonstrates the effectiveness of the proposed method.

7. An attention-based end-to-end model for multiple text lines recognition in japanese historical documents

   Nam Tuan Ly, Cuong Tuan Nguyen, and Masaki Nakagawa

   In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR 2019, Sep 2019

   Abs

   This paper presents an attention-based convolutional sequence to sequence (ACseq2seq) model for recognizing an input image of multiple text lines from Japanese historical documents without explicit segmentation of lines. The recognition system has three main parts: a feature extractor using Convolutional Neural Network (CNN) to extract a feature sequence from an input image; an encoder employing bidirectional Long Short-Term Memory (BLSTM) to encode the feature sequence; and a decoder using a unidirectional LSTM with the attention mechanism to generate the final target text based on the attended pertinent features. We also introduce a residual LSTM network between the attention vector and softmax layer in the decoder. The system can be trained end-to-end by a standard cross-entropy loss function. In the experiment, we evaluate the performance of the ACseq2seq model on the anomalously deformed Kana datasets in the PRMU contest. The results of the experiments show that our proposed model achieves higher recognition accuracy than the state-of-the-art recognition methods on the anomalously deformed Kana datasets.

2018

1. A database of unconstrained Vietnamese online handwriting and recognition experiments by recurrent neural networks

   Hung Tuan Nguyen, Cuong Tuan Nguyen, Pham The Bao, and 1 more author

   Pattern Recognition, Sep 2018

   Abs

   We present our efforts to create a database of unconstrained Vietnamese online handwritten text sampled from pen-based devices. The database stores handwritten text for paragraphs, lines, words, and characters, with the ground truth associated with every paragraph and line. We show a detailed statistical analysis of the handwritten text in this database and describe recognition experiments using several recent methods including the Bidirectional Long Short-Term Memory (BLSTM) network. Overall, our database contains over 480,000 strokes from more than 380,000 characters, which, at present, is the largest database of Vietnamese online handwritten text. Although Vietnamese script is based on a fixed set of alphabet letters, the recognition of Vietnamese online handwritten text poses a difficult challenge because of many diacritical marks, which usually result in delayed strokes during writing. We designed and implemented an online handwriting-collection tool to gather data, as well as a line-segmentation tool and a delayed-stroke-detection tool to analyze collected handwritten text. We also conducted a statistical analysis based on the writer profiles. We applied a number of the state-of-the-art recognition methods on unconstrained Vietnamese handwriting to evaluate their performance, including the BLSTM network, which is an efficient architecture derived from the Recurrent Neural Network (RNN) and is often applied to sequence labeling problems. The BLSTM network achieved 90% character recognition accuracy, despite many long sequences with several delayed strokes. Our database is allowed open access for research to stimulate the development of handwriting research technology.

2. Training an end-to-end model for offline handwritten Japanese text recognition by generated synthetic patterns

   Nam Tuan Ly, Cuong Tuan Nguyen, and Masaki Nakagawa

   In Proceedings of International Conference on Frontiers in Handwriting Recognition, ICFHR 2018, Sep 2018

   Abs

   This paper presents an end-to-end model of Deep Convolutional Recurrent Network (DCRN) for recognizing offline handwritten Japanese text lines. The end-to-end DCRN model has three parts: A convolutional feature extractor using Deep Convolutional Neural Network (DCNN) to extract a feature sequence from a text line image; recurrent layers employing a Deep Bidirectional LSTM to predict pre-frame from the feature sequence; and a transcription layer using Connectionist Temporal Classification (CTC) to convert the pre-frame predictions into the label sequence. Since our end-to-end model requires a large data for training, we synthesize handwritten text line images from sentences in corpora and handwritten character patterns in the Nakayosi and Kuchibue database with elastic distortions. In the experiment, we evaluate the performance of the end-to-end model and the effectiveness of the synthetic data generation method on the test set of the TUAT Kondate database. The results of the experiments show that our end-to-end model achieves higher than the state-of-the-art recognition accuracy on the test set of TUAT Kondate with 96.35% and 98.05% character level recognition accuracies without and with the generated synthetic data, respectively.

3. ICFHR 2018-Competition on Vietnamese online handwritten text recognition using HANDS-VNOnDB (VOHTR2018)

   Hung Tuan Nguyen, Cuong Tuan Nguyen, and Masaki Nakagawa

   In Proceedings of International Conference on Frontiers in Handwriting Recognition, ICFHR 2018, Sep 2018

   Abs

   This paper presents the results of the VOHTR 2018 competition on Vietnamese Online Handwritten Text Recognition. The goal of this competition is to evaluate and compare recent online handwritten text recognition systems on Vietnamese online handwritten text which contains many delayed strokes caused by the diacritic marks. Besides, the general objective is to encourage the studies on Vietnamese online handwritten text recognition based on the large Vietnamese handwriting database collected from 200 writers. In this competition, we introduce three tasks consisting of word recognition (task 1), text-line recognition (task 2) and paragraph recognition (task 3) which are described in details. Subsequently, we describe the evaluation metrics and give comparative results of competitors along with the brief descriptions of the respective methods.

4. Clustering Offline Handwritten Mathematical Answers for Computer-Assisted Marking

   Vu Tran Minh Khuong, Huy Quang Ung, Cuong Tuan Nguyen, and 1 more author

   In Proceedings of the International Conference on Pattern Recognition and Machine Intelligent, ICPRAI 2018, Sep 2018
5. Online Japanese handwriting recognizers using recurrent neural networks

   Hung Tuan Nguyen, Cuong Tuan Nguyen, and Masaki Nakagawa

   In Proceedings of International Conference on Frontiers in Handwriting Recognition, ICFHR 2018, Sep 2018

   Abs

   This paper presents an attempt to recognize online isolated handwritten Japanese characters as well as text lines by Recurrent Neural Networks (RNNs). Although there are some successful studies on online Chinese handwriting recognition using RNNs, it is difficult to achieve high accuracy for Japanese due to many different types of characters such as kanji and kana. Moreover, training RNNs usually requires a large number of samples for each class of character. Hence, we apply five different transformation operations on the original samples to generate artificial samples with various deformations. For online handwritten Japanese text, we use the Kondate database, but it does not cover the whole Japanese character set. Thus, we generate random text lines using the sentences from corpora and isolated characters from the Nakayosi and Kuchibue databases. Besides, we employ a robust process with some preprocessing steps and the state-of-the-art online features to extract invariant features from handwritten patterns because the handwritten samples are merged from the different databases collected on different devices with various resolutions. For the recognition model, we have implemented the different Bidirectional Long Short-Term Memory Networks (BLSTM networks) for isolated character classification (sequence classification task) and handwritten text recognition (transcription task). The best model of the sequence classification task achieves an accuracy of 97.91% on Nakayosi and 97.74% on Kuchibue. The best model of transcription task performs a character recognition rate of 86.31% on Kondate and 83.15% on the generated text lines.

2017

1. A segmentation method of single- and multiple-touching characters in offline handwritten Japanese text recognition

   Kha Cong Nguyen, Cuong Tuan Nguyen, and Masaki Nakagawa

   IEICE Transactions on Information and Systems, Sep 2017

   Abs

   This paper presents a method to segment single- and multiple-touching characters in offline handwritten Japanese text recognition with practical speed. Distortions due to handwriting and a mix of complex Chinese characters with simple phonetic and alphanumeric characters leave optical handwritten text recognition (OHTR) for Japanese still far from perfection. Segmentation of characters, which touch neighbors on multiple points, is a serious unsolved problem. Therefore, we propose a method to segment them which is made in two steps: coarse segmentation and fine segmentation. The coarse segmentation employs vertical projection, stroke-width estimation while the fine segmentation takes a graph-based approach for thinned text images, which employs a new bridge finding process and Voronoi diagrams with two improvements. Unlike previous methods, it locates character centers and seeks segmentation candidates between them. It draws vertical lines explicitly at estimated character centers in order to prevent vertically unconnected components from being left behind in the bridge finding. Multiple candidates of separation are produced by removing touching points combinatorially. SVM is applied to discard improbable segmentation boundaries. Then, ambiguities are finally solved by the text recognition employing linguistic context and geometric context to recognize segmented characters. The results of our experiments show that the proposed method can segment not only single-touching characters but also multiple-touching characters, and each component in our proposed method contributes to the improvement of segmentation and recognition rates.

2. Attempts to recognize anomalously deformed Kana in Japanese historical documents

   Hung Tuan Nguyen, Nam Tuan Ly, Kha Cong Nguyen, and 2 more authors

   In Proceedings of the 4th International Workshop on Historical Document Imaging and Processing, HIP 2017, Sep 2017

   Abs

   This paper presents methods for three different tasks of recognizing anomalously deformed Kana in Japanese historical documents, which were contested by IEICE PRMU1 2017. The tasks have three levels: single character recognition, three Kana characters sequence recognition and unrestricted Kana recognition. We compare several methods for each task. For the level 1, we evaluate CNN based methods and BLSTM based methods. For the level 2, we consider several variations of a combined architecture of CNN and BLSTM. For the level 3, we compare an extension of the method for the level 2 and a segmentation based method. We achieve the single character recognition accuracy of 96.8%, the three Kana characters sequence recognition accuracy of 87.12% and the unrestricted Kana recognition accuracy of 73.3%. These results prove the performance of CNN and BLSTM on these tasks.

3. Tens of thousands of nom character recognition by deep convolution neural networks

   Cong Kha Nguyen, Cuong Tuan Nguyen, and Nakagawa Masaki

   In Proceedings of the 4th International Workshop on Historical Document Imaging and Processing, HIP 2017, Sep 2017

   Abs

   This paper proposes a method to recognize a large set of 32,695 Nom characters which had been used in Vietnam from the tenth century to the twentieth century before the Latin-based Vietnamese alphabet became common. So far, the largest sets to which character recognition methods have been studied, including the latest deep Neural Networks, are about 10,000 for the current set of Chinese, Japanese and Korean, but ancient languages of Chinese origin require much larger sizes of categories. Moreover, lack of training patterns makes the development of Optical Character Recognition (OCR) for Nom be a big challenge. On the other hand, the demand to archive Nom historical documents is very high since a large amount of documents are left uninterpreted and scholars who can comprehend Nom are decreasing. Therefore, we propose a method to recognize a very large set of Nom categories by Deep Convolution Neural Networks (CNN). The proposed method introduces coarse categories which are prepared by K-means beforehand. We construct deep CNNs composed by a coarse category feature extractor, a coarse category classifier and a fine category classifier including some inception modules. We pre-train the coarse category feature extractor and the coarse category classifier with the coarse categories, freeze them and then perform fine tuning to recognize characters in the whole categories of Nom. Unlike conventional cascade of coarse classification and fine classification, the coarse and fine category classifiers are executed in parallel to feature maps generated by the feature extractor and their likelihoods are multiplied. The experiment shows that this architecture provides the better recognition rate than the former cascade of GLVQ and MQDF.