Publications | Cuong Tuan Nguyen

2022

PRL

Syntactic data generation for handwritten mathematical expression recognition

Thanh Nghia Truong, Cuong Tuan Nguyen, and Masaki Nakagawa

Pattern Recognition Letters, 2022

Abs

This paper proposes tree-based decomposition and sub-expression interchange for generating new syntactically valid handwritten mathematical expressions (HMEs) from a given set of HMEs to train an HME recognition model and a mathematical language model (LM). The recognition model is dual trained using weakly supervised learning and encoder-decoder attention loss on the generated samples. Recognition experiments indicate that the proposed data generation method is superior to other such methods for offline HMEs. The HME recognition model increases the expression recognition rates by 1.47, 2.88, and 2.67 points on the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014, 2016, and 2019 testing sets, respectively. The LM also increases them by 8.92, 6.88, and 2.59 points on the testing sets. Further adding extra LaTeX sequences is cost effective in strengthening the LM for the expression recognition rates being 2.54, 2.8, and 1.25 points higher than without them on the CROHME testing sets, respectively. Among academic systems, the trained HME recognition system achieves the best performance with 64.60% and 66.08% expression recognition rates on the CROHME 2014 and 2016 testing sets and a comparable expression recognition rate of 58.72% on the CROHME 2019 testing set, respectively. Comparison with top systems from companies on CROHME 2019 suggests that more real and/or generated HME patterns will improve the performance of HME recognition models as well as mathematical language models.
Learning Symbol Relation Tree for Online Handwritten Mathematical Expression Recognition

Thanh Nghia Truong, Hung Tuan Nguyen, Cuong Tuan Nguyen, and 1 more author

In Proceedings - 6th IAPR Asian Conference on Pattern Recognition, ACPR 2021, 2022

Abs

This paper proposes a method for recognizing online handwritten mathematical expressions (OnHME) by building a symbol relation tree (SRT) directly from a sequence of strokes. The recognition system has two parts: a temporal classifier and a tree connector. The temporal classifier uses global context to produce a sequence of symbols and spatial relations between symbols from an OnHME pattern. It is a bidirectional recurrent neural network trained from multiple derived paths of SRTs. The tree connector splits the sequence into several sub-SRTs and connects them to form the SRT by looking up the best combination among those sub-SRTs. Besides, we adopt a tree sorting method to deal with various stroke orders. Recognition experiments indicate that the proposed OnHME recognition system is competitive to other methods. The recognition system achieves 44.12% and 41.76% expression recognition rates on the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014 and 2016 testing sets, respectively.
Fully Automated Short Answer Scoring of the Trial Tests for Common Entrance Examinations for Japanese University

Haruki Oka, Hung Tuan Nguyen, Cuong Tuan Nguyen, and 2 more authors

In Proceedings - 23rd International Conference Artificial Intelligence in Education, AIED 2022, 2022

Abs

Studies on automated short-answer scoring (SAS) have been conducted to apply natural language processing to education. Short-answer scoring is a task to grade the responses from linguistic information. Most answer sheets for short-answer questions are handwritten in an actual educational setting, which is a barrier to SAS. Therefore, we have developed a system that uses handwritten character recognition and natural language processing for fully automated scoring of handwritten responses to short-answer questions. This is the most extensive scoring data for responses to short-answer questions, and it may be the largest in the world. Applying the Cohen’s kappa coefficient to the graded evaluations, the results show 0.86 in the worst case, and approximately 0.95 is recorded for the remaining five question answers. We observe that the fully automated scoring system proposed in our study can also score with a high degree of accuracy comparable to that of human scoring.

2021

PRL

Clustering online handwritten mathematical expressions

Huy Quang Ung, Cuong Tuan Nguyen, Khanh Minh Phan, and 2 more authors

Pattern Recognition Letters, Jun 2021

Abs

To help human markers mark many answers in the form of online handwritten mathematical expressions (OHMEs), this paper proposes bag-of-features for clustering OHMEs. It consists of six levels of features from low-level pattern features to high-level symbolic and structural features obtained from a state-of-the-art OHME recognizer. Then, it introduces distance-based representation (DbR) to reduce the dimensionality of our proposed feature spaces. Moreover, it presents a method for combining the proposed features to improve the performance. Experiments using the k-means++ algorithm are conducted on a set of 3,150 OHMEs (Dset_50) and an answer dataset (Dset_Mix) of 200 OHMEs intermixed between real patterns and synthesized patterns for each of 10 questions. When the number of clusters is set as the true number of categories, the best purity around 0.99 is produced by bag-of-symbols with DbR for Dset_50, which is better than state-of-the-art methods for clustering offline patterns converted from their OHMEs. The combination of both low-level and high-level features with DbR achieves a purity of around 0.777, increases to more than 0.90 and reduce the marking cost by more than 0.35 point than manually marking OHME answers by adjusting the number of clusters for Dset_Mix.
Clustering of handwritten mathematical expressions for computer-assisted marking

Vu Tran Minh KHUONG, Khanh Minh PHAN, Huy Quang UNG, and 2 more authors

IEICE Transactions on Information and Systems, Jun 2021

Abs

Many approaches enable teachers to digitalize students’ answers and mark them on the computer. However, they are still limited for supporting marking descriptive mathematical answers that can best evaluate learners’ understanding. This paper presents clustering of offline handwritten mathematical expressions (HMEs) to help teachers efficiently mark answers in the form of HMEs. In this work, we investigate a method of combining feature types from low-level directional features and multiple levels of recognition: Bag-of-symbols, bag-of-relations, and bag-ofpositions. Moreover, we propose a marking cost function to measure the marking effort. To show the effectiveness of our method, we used two datasets and another sampled from CROHME 2016 with synthesized patterns to prepare correct answers and incorrect answers for each question. In experiments, we employed the k-means++ algorithm for each level of features and considered their combination to produce better performance. The experiments show that the best combination of all the feature types can reduce the marking cost to about 0.6 by setting the number of answer clusters appropriately compared with the manual one-by-one marking.
ICDAR-W

A Transformer-Based Math Language Model for Handwritten Math Expression Recognition

Huy Quang Ung, Cuong Tuan Nguyen, Hung Tuan Nguyen, and 2 more authors

In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR 2021 Workshops, Jun 2021

Abs

Handwritten mathematical expressions (HMEs) contain ambiguities in their interpretations, even for humans sometimes. Several math symbols are very similar in the writing style, such as dot and comma or “0”, “O”, and “o”, which is a challenge for HME recognition systems to handle without using contextual information. To address this problem, this paper presents a Transformer-based Math Language Model (TMLM). Based on the self-attention mechanism, the high-level representation of an input token in a sequence of tokens is computed by how it is related to the previous tokens. Thus, TMLM can capture long dependencies and correlations among symbols and relations in a mathematical expression (ME). We trained the proposed language model using a corpus of approximately 70,000 LaTeX sequences provided in CROHME 2016. TMLM achieved the perplexity of 4.42, which outperformed the previous math language models, i.e., the N-gram and recurrent neural network-based language models. In addition, we combine TMLM into a stochastic context-free grammar-based HME recognition system using a weighting parameter to re-rank the top-10 best candidates. The expression rates on the testing sets of CROHME 2016 and CROHME 2019 were improved by 2.97 and 0.83 percentage points, respectively.
ICDAR

Global Context for Improving Recognition of Online Handwritten Mathematical Expressions

Cuong Tuan Nguyen, Thanh Nghia Truong, Hung Tuan Nguyen, and 1 more author

In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR 2021, Jun 2021

Abs

This paper presents a temporal classification method for all three subtasks of symbol segmentation, symbol recognition and relation classification in online handwritten mathematical expressions (HMEs). The classification model is trained by multiple paths of symbols and spatial relations derived from the Symbol Relation Tree (SRT) representation of HMEs. The method benefits from global context of a deep bidirectional Long Short-term Memory network, which learns the temporal classification directly from online handwriting by the Connectionist Temporal Classification loss. To recognize an online HME, a symbol-level parse tree with Context-Free Grammar is constructed, where symbols and spatial relations are obtained from the temporal classification results. We show the effectiveness of the proposed method on the two latest CROHME datasets.
ICDAR

GSSF: A Generative Sequence Similarity Function Based on a Seq2Seq Model for Clustering Online Handwritten Mathematical Answers

Huy Quang Ung, Cuong Tuan Nguyen, Hung Tuan Nguyen, and 1 more author

In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR 2021, Jun 2021

Abs

Toward a computer-assisted marking for descriptive math questions, this paper presents clustering of online handwritten mathematical expressions (OnHMEs) to help human markers to mark them efficiently and reliably. We propose a generative sequence similarity function for computing a similarity score of two OnHMEs based on a sequence-to-sequence OnHME recognizer. Each OnHME is represented by a similarity-based representation (SbR) vector. The SbR matrix is inputted to the k-means algorithm for clustering OnHMEs. Experiments are conducted on an answer dataset (Dset_Mix) of 200 OnHMEs mixed of real patterns and synthesized patterns for each of 10 questions and a real online handwritten mathematical answer dataset of 122 student answers at most for each of 15 questions (NIER_CBT). The best clustering results achieved around 0.916 and 0.915 for purity, and around 0.556 and 0.702 for the marking cost on Dset_Mix and NIER_CBT, respectively. Our method currently outperforms the previous methods for clustering HMEs.
ICDAR-W

Relation-Based Representation for Handwritten Mathematical Expression Recognition

Thanh Nghia Truong, Huy Quang Ung, Hung Tuan Nguyen, and 2 more authors

In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR 2021 Workshops, Jun 2021

Abs

This paper proposes relation-based sequence representation that enhances offline handwritten mathematical expressions (HMEs) recognition. Commonly, a LaTeX-based sequence represents the 2D structure of an HME as a 1D sequence. Consequently, the LaTeX-based sequence becomes longer, and HME recognition systems have difficulty in extracting its 2D structure. We propose a new representation for HMEs according to the relations of symbols, which shortens the LaTeX-based representation. We use an offline end-to-end HME recognition system that adopts weakly supervised learning to evaluate the proposed representation. Recognition experiments indicate that the proposed relation-based representation helps the HME recognition system achieve higher performance than the LaTeX-based representation. In fact, the HME recognition system achieves recognition rates of 53.35%, 52.14%, and 53.13% on the dataset of the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014, 2016, and 2019, respectively. These results are more than 2 percentage points higher than the LaTeX-based system.
ICDAR-W

Temporal Classification Constraint for Improving Handwritten Mathematical Expression Recognition

Cuong Tuan Nguyen, Hung Tuan Nguyen, Kei Morizumi, and 1 more author

In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR 2021 Workshops, Jun 2021

Abs

We present a temporal classification constraint as an auxiliary learning method for improving the recognition of Handwritten Mathematical Expression (HME). Connectionist temporal classification (CTC) is used to learn the temporal alignment of the input feature sequence and corresponding symbol label sequence. The CTC alignment is trained with the encoder-decoder alignment through a combination of CTC loss and encoder-decoder loss to improve the feature learning of the encoder in the encoder-decoder model. We show the effectiveness of the approach in improving symbol classification and expression recognition on the CROHME datasets.

2020

PRL

An attention-based row-column encoder-decoder model for text recognition in Japanese historical documents

Nam Tuan Ly, Cuong Tuan Nguyen, and Masaki Nakagawa

Pattern Recognition Letters, Jun 2020

Abs

This paper presents an attention-based row-column encoder-decoder (ARCED) model for recognizing an input image of multiple text lines from Japanese historical documents without explicit segmentation of lines. The recognition system has three main parts: a feature extractor, a row-column encoder, and a decoder. We introduce a row-column BLSTM in the encoder and a residual LSTM network in the decoder. The whole system is trained end-to-end by a standard cross-entropy loss function, requiring only document images and their ground-truth text. We experimentally evaluate the performance of ARCED on the dataset of Japanese historical documents: Kana-PRMU. The results of the experiments show that ARCED outperforms the state-of-the-art recognition methods on the dataset. Furthermore, we demonstrate that the row-column BLSTM in the encoder and the residual LSTM in the decoder improves the performance of the encoder-decoder model for the recognition of Japanese historical document.
A unified method for augmented incremental recognition of online handwritten Japanese and English text

Cuong Tuan Nguyen, Bipin Indurkhya, and Masaki Nakagawa

International Journal on Document Analysis and Recognition, Jun 2020

Abs

We present a unified method to augmented incremental recognition for online handwritten Japanese and English text, which is used for busy or on-the-fly recognition while writing, and lazy or delayed recognition after writing, without incurring long waiting times. It extends the local context for segmentation and recognition to a range of recent strokes called “segmentation scope” and “recognition scope,” respectively. The recognition scope is inside of the segmentation scope. The augmented incremental recognition triggers recognition at every several recent strokes, updates the segmentation and recognition candidate lattice, and searches over the lattice for the best result incrementally. It also incorporates three techniques. The first is to reuse the segmentation and recognition candidate lattice in the previous recognition scope for the current recognition scope. The second is to fix undecided segmentation points if they are stable between character/word patterns. The third is to skip recognition of partial candidate character/word patterns. The augmented incremental method includes the case of triggering recognition at every new stroke with the above-mentioned techniques. Experiments conducted on TUAT-Kondate and IAM online database show its superiority to batch recognition (recognizing text at one time) and pure incremental recognition (recognizing text at every input stroke) in processing time, waiting time, and recognition accuracy.
PRL

Nom document digitalization by deep convolution neural networks

Kha Cong Nguyen, Cuong Tuan Nguyen, and Masaki Nakagawa

Pattern Recognition Letters, Jun 2020

Abs

Nom is an ancient script used in Vietnam until the current Latin-based Vietnamese alphabet became common, and a large number of ancient Nom documents are in existence. Due to the gradual degradation of Nom documents and a decrease in the number of scholars who can understand them, a system to digitalize Nom documents is urgently necessary. This paper presents a segmentation-based method for digitalizing Nom documents using deep convolution neural networks. Nom pages are preprocessed, segmented into isolated characters, and then recognized by a single-character OCR. The structure of the U-Net is applied to create segmentation maps and extract character regions from them. Subsequently, we propose coarse and fine combined classifiers to recognize each character pattern. The results by the best classifier are revised by a decoder using a langue model. The decoder is the same as the connectionist temporal classification decoder used in end-to-end text recognition systems. Compared with the traditional segmentation method using projection profiles and the Voronoi diagram (IoU = 81.23%), the segmentation method using the deep convolution neural network produces a better result (IoU = 92.08%) for detecting character regions. The proposed CNN models for recognizing segmented character patterns outperforms the traditional models using the modified quadratic discriminant function and the learning vector quantization with the recognition rate of 85.07%. The combination of coarse and fine classifiers, the training dataset with salt and pepper noises, and the attention layer are the key factors in the recognition rate improvement.
PRL

CNN based spatial classification features for clustering offline handwritten mathematical expressions

Cuong Tuan Nguyen, Vu Tran Minh Khuong, Hung Tuan Nguyen, and 1 more author

Pattern Recognition Letters, Jun 2020

Abs

To help human markers mark a large number of answers of handwritten mathematical expressions (HMEs), clustering them makes marking more efficient and reliable. Clustering HMEs, however, faces the problem of extracting both localization and classification representation of mathematical symbols for an HME image and defining the distance between two HME images. First, we propose a method based on Convolutional Neural Networks (CNN) to extract the representations for an HME. Symbols in various scales are located and classified by a combination of features from a multi-scale CNN. We use weakly supervised training combined with symbols attention to enhance localization and classification predictions. Second, we propose a multi-level spatial distance between two representations for clustering HMEs. Experiments on CROHME 2016 and CROHME 2019 dataset show the promising results of 0.99 and 0.96 in purity, respectively.
ICFHR

A Semantic Segmentation-based Method for Handwritten Japanese Text Recognition

Kha Cong Nguyen, Cuong Tuan Nguyen, and Masaki Nakagawa

In Proceedings of International Conference on Frontiers in Handwriting Recognition, ICFHR 2020, Jun 2020

Abs

Recently, the segmentation-free approach using Convolution Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) for handwritten text recognition has been investigated by many research groups. Although good results are produced on some datasets, there remain some drawbacks. It is neither robust against the change of handwriting styles such as gaps between characters, stroke widths and shapes of characters nor stable for skewed, and curved text lines unless heavily trained by such patterns. In this paper, we propose a segmentation-based method for handwritten Japanese text recognition. The method employs a semantic segmentation model for precisely splitting text lines into single characters. The semantic segmentation model is based on an encoder-decoder architecture like U-Net, but we employ available techniques to improve the accuracy of pixel classification. They are a deeper encoder with ResNet 101, dilated convolutions and Spatial Pyramid Pooling. Subsequently, a CNN based OCR is used to recognize segmented character images. Finally, the recognized candidates are considered in a lattice diagram combined with a linguistic context. The result of experiments shows that the segmentation model increases the mean IoU significantly from 89.32% to 94.96% and the proposed approach is robust to the change of handwriting styles while a segmentation-free method is quite sensitive to them, resulting in the significant reduction of the recognition rate.
ICFHR

A Siamese Network-based Approach for Matching Various Sizes of Excavated Wooden Fragments

Trung Tan Ngo, Cuong Tuan Nguyen, and Masaki Nakagawa

In Proceedings of International Conference on Frontiers in Handwriting Recognition, ICFHR 2020, Jun 2020

Abs

This paper presents an approach for matching various sizes of excavated wooden fragments based on siamese neural networks. We propose a siamese network composed of global average pooling and a fine-tuned Resnet encoder (GA-S-net). We also propose an elaborated siamese network by replacing the global average pooling with a spatial pyramid pooling layer and add a new dense absolute difference layer (SP-S-net). Samples of 37,760 fragments were prepared from 268 complete wooden tablets excavated from the Heijo-Kyo Palace ruins used during the Nara period in Japan. Both of the networks answer whether two fragments are from the same tablet or not. The result of both networks for the testing set is similar to AUC (Area under the curve) of ROC (Receiver Operating Characteristic) curve being around 90%. In AUC of large fragments, however, SP-S-net is better than GA-S-net with 97.1% versus 93.8%. These networks are rather new for dealing with various sizes of inputs for the matching problem.
ICFHR

Attention Augmented Convolutional Recurrent Network for Handwritten Japanese Text Recognition

Nam Tuan Ly, Cuong Tuan Nguyen, and Masaki Nakagawa

In Proceedings of International Conference on Frontiers in Handwriting Recognition, ICFHR 2020, Jun 2020

Abs

Handwritten Japanese text recognition is still a big challenging task due to the large character set, diversity of writing styles, and multiple-touches between characters. In this paper, we propose a model of Attention Augmented Convolutional Recurrent Network (AACRN) for recognizing handwritten Japanese text lines. The AACRN model has three main parts: A convolutional feature extractor, a self-attention based encoder, and a CTC-decoder. The whole model can be trained end-to-end. In the experiment, we evaluate the performance of the AACRN model on the TUAT Kondate dataset and the Kuzushiji dataset. The results of the experiments show that the proposed model achieves higher performance than the state-of-the-art recognition accuracies on the test set of TUAT Kondate and the Kuzushiji dataset.
ICPR

Online trajectory recovery from offline handwritten Japanese kanji characters of multiple strokes

Hung Tuan Nguyen, Tsubasa Nakamura, Cuong Tuan Nguyen, and 1 more author

In Proceedings - International Conference on Pattern Recognition, ICPR 2020, Jun 2020

Abs

We propose a deep neural network-based method to recover dynamic online trajectories from offline handwritten Japanese kanji character images. It is a challenging task since Japanese kanji characters consist of multiple strokes. Our proposed model has three main components: Convolutional Neural Network-based encoder, Long Short-Term Memory Network-based decoder with an attention layer, and Gaussian Mixture Model (GMM). The encoder focuses on feature extraction while the decoder refers to the extracted features and generates time-sequences of GMM parameters. The attention layer is the key component for trajectory recovery. The GMM provides robustness to style variations so that the proposed model does not overfit to training samples. In the experiments, the proposed method is evaluated by both visual verification and handwritten character recognition. This is the first attempt to use online recovered trajectories to help improve offline handwriting recognition performance. Although the visual verification reveals some problems, the recognition experiments demonstrate the effect of trajectory recovery in improving offline handwritten character recognition accuracy when online recognition of the recovered trajectories are combined.

2019

An online overlaid handwritten Japanese text recognition system for small tablet

Jianjuan Liang, Cuong Tuan Nguyen, Bilan Zhu, and 1 more author

Pattern Analysis and Applications, Jun 2019

Abs

The paper presents a recognition system of online overlaid handwritten Japanese text patterns on a smart phone or baby-face tablet. The proposed system oversegments a sequence of strokes into primitive segments at candidate off-strokes between strokes using a SVM model. One or more consecutive primitive segments form a candidate character pattern, which is recognized into a list of candidate categories. Then, a segmentation and recognition candidate lattice is constructed to represent all candidate character patterns and their corresponding character classes. Finally, the optimal path is effectively found by the Viterbi search in the lattice, combining the scores of character recognition, geometric features, linguistic context, as well as the segmentation scores by SVM classification. This system incorporates feature reduction and non-character pruning to decrease the time cost per character, and semi-incremental recognition to decrease waiting time. The recognition rates on generated and collected overlaid handwritten text are 92.16% and 93.04%, respectively. The average time cost per character is not more than 0.6 s, and the average waiting time is less than 0.875 s even on an Intel Atom 1.33 GHz CPU: a low power consumption CPU for small tablets and embedded devices. Therefore, we confirm that our system recognizes online overlaid handwritten text composed of thousands of Japanese character classes with the high recognition rate without excessive waiting time.
PRL

Robust and real-time stroke order evaluation using incremental stroke context for learners to write Kanji characters correctly

Cuong Tuan Nguyen, Hung Tuan Nguyen, Kazuhiro Mita, and 1 more author

Pattern Recognition Letters, Apr 2019

Abs

Writing Kanji characters of Chinese origin in the correct stroke order and direction is still one of the important subjects in Japanese elementary education. So far, the stroke order evaluation was made by stroke-to-stroke matching without stroke context so that it was unrobust to Kanji characters having multiple similar strokes. Here, we employ shape context features around each feature point in not only conventional fan-shaped bins but also in square bins with applying a Gaussian function. We also propose simple incremental context and augmented context from future strokes. Our approach can judge whether the stroke order and direction are correct or not every time a new stroke is written on a tablet by matching a partially written Kanji pattern with the reference pattern written to the same number of strokes. Evaluation shows that the best-tuned method with square bins and the Gaussian function records the highest performance and correctly evaluates stroke order by 98.5% with the maximum time of 0.12 sec. /character for Kanji patterns after all strokes are written using an average desktop PC. The method is also shown to possess high reliability to detect wrong stroke order and direction incrementally every time after each stroke is written.
Recognition of anomalously deformed kana sequences in Japanese historical documents

Nam Tuan Ly, Kha Cong Nguyen, Cuong Tuan Nguyen, and 1 more author

IEICE Transactions on Information and Systems, Apr 2019

Abs

This paper presents recognition of anomalously deformed Kana sequences in Japanese historical documents, for which a contest was held by IEICE PRMU 2017. The contest was divided into three levels in accordance with the number of characters to be recognized: Level 1: Single characters, level 2: Sequences of three vertically written Kana characters, and level 3: Unrestricted sets of characters composed of three or more characters possibly in multiple lines. This paper focuses on the methods for levels 2 and 3 that won the contest. We basically follow the segmentationfree approach and employ the hierarchy of a Convolutional Neural Network (CNN) for feature extraction, Bidirectional Long Short-Term Memory (BLSTM) for frame prediction, and Connectionist Temporal Classification (CTC) for text recognition, which is named a Deep Convolutional Recurrent Network (DCRN). We compare the pretrained CNN approach and the end-to-end approach with more detailed variations for level 2. Then, we propose a method of vertical text line segmentation and multiple line concatenation before applying DCRN for level 3. We also examine a twodimensional BLSTM (2DBLSTM) based method for level 3. We present the evaluation of the best methods by cross validation. We achieved an accuracy of 89.10% for the three-Kana-character sequence recognition and an accuracy of 87.70% for the unrestricted Kana recognition without employing linguistic context. These results prove the performances of the proposed models on the level 2 and 3 tasks.
PRL

Text-independent writer identification using convolutional neural network

Hung Tuan Nguyen, Cuong Tuan Nguyen, Takeya Ino, and 2 more authors

Pattern Recognition Letters, Apr 2019

Abs

The text-independent approach to writer identification does not require the writer to write some predetermined text. Previous research on text-independent writer identification has been based on identifying writer-specific features designed by experts. However, in the last decade, deep learning methods have been successfully applied to learn features from data automatically. We propose here an end-to-end deep-learning method for text-independent writer identification that does not require prior identification of features. A Convolutional Neural Network (CNN) is trained initially to extract local features, which represent characteristics of individual handwriting in the whole character images and their sub-regions. Randomly sampled tuples of images from the training set are used to train the CNN and aggregate the extracted local features of images from the tuples to form global features. For every training epoch, the process of randomly sampling tuples is repeated, which is equivalent to a large number of training patterns being prepared for training the CNN for text-independent writer identification. We conducted experiments on the JEITA-HP database of offline handwritten Japanese character patterns. With 200 characters, our method achieved an accuracy of 99.97% to classify 100 writers. Even when using 50 characters for 100 writers or 100 characters for 400 writers, our method achieved accuracy levels of 92.80% or 93.82%, respectively. We conducted further experiments on the Firemaker and IAM databases of offline handwritten English text. Using only one page per writer to train, our method achieved over 91.81% accuracy to classify 900 writers. Overall, we achieved a better performance than the previously published best result based on handcrafted features and clustering algorithms, which demonstrates the effectiveness of our method for handwritten English text also.
ICDAR

A character attention generative adversarial network for degraded historical document restoration

Kha Cong Nguyen, Cuong Tuan Nguyen, Seiji Hotta, and 1 more author

Apr 2019

Abs

Despite of recent breakthroughs in the accuracy of single character recognition using the deeper convolution neural networks, one of the remaining problems is that OCRs almost fail to recognize character patterns when they are severely degraded, especially those of the historical documents. Another problem to recognize characters in historical documents is the lack of sufficient training patterns because of the heavy cost for annotation. This paper proposes a character attention generative adversarial network named CAGAN for restoring heavily degraded character patterns in historical documents so that OCRs improve their accuracy and even help archeologists to decode them. The network is based on the U-Net like architecture [1] with skip connections, and it is trained by the proposed loss function including the common adversarial loss (global loss) and the hierarchical character attentive loss (local loss). We made an experiment on 118 categories of most common Japanese Kanji characters, collected from severely damaged historical documents called Heijokyo mokkan written during the Nara period in Japan. The experiment shows that our method restores the shapes of characters and improves the recognition rate significantly, which is helpful for archeologists to decode damaged character patterns.
An attention-based end-to-end model for multiple text lines recognition in japanese historical documents

Nam Tuan Ly, Cuong Tuan Nguyen, and Masaki Nakagawa

Apr 2019

Abs

This paper presents an attention-based convolutional sequence to sequence (ACseq2seq) model for recognizing an input image of multiple text lines from Japanese historical documents without explicit segmentation of lines. The recognition system has three main parts: a feature extractor using Convolutional Neural Network (CNN) to extract a feature sequence from an input image; an encoder employing bidirectional Long Short-Term Memory (BLSTM) to encode the feature sequence; and a decoder using a unidirectional LSTM with the attention mechanism to generate the final target text based on the attended pertinent features. We also introduce a residual LSTM network between the attention vector and softmax layer in the decoder. The system can be trained end-to-end by a standard cross-entropy loss function. In the experiment, we evaluate the performance of the ACseq2seq model on the anomalously deformed Kana datasets in the PRMU contest. The results of the experiments show that our proposed model achieves higher recognition accuracy than the state-of-the-art recognition methods on the anomalously deformed Kana datasets.
Deep Convolutional Recurrent Network for Segmentation-Free Offline Handwritten Japanese Text Recognition

Nam Tuan Ly, Cuong Tuan Nguyen, Kha Cong Nguyen, and 1 more author

Apr 2019

Abs

This paper presents a model of Deep Convolutional Recurrent Network (DCRN) for recognizing offline handwritten Japanese text lines without explicit segmentation of characters. Most of traditional offline handwritten Japanese text recognizers perform segmentation of text image into characters before individually recognizing each character. Although segmentation by recognition and context are employed to recover from segmentation errors, errors made at this stage directly make an impact on the performance of the whole system. The DCRN model consists of three parts: a convolutional feature extractor using Convolutional Neural Network (CNN) and sliding window to extract features from text image; recurrent layers using BLSTM to predict pre-frame from an input sequence; and a transcription layer using a CTC-decoder to translate the predictions into the label sequence. Experimental results on the database: TUAT Kondate database demonstrates the effectiveness of the proposed method.

2018

PR

A database of unconstrained Vietnamese online handwriting and recognition experiments by recurrent neural networks

Hung Tuan Nguyen, Cuong Tuan Nguyen, Pham The Bao, and 1 more author

Pattern Recognition, Apr 2018

Abs

We present our efforts to create a database of unconstrained Vietnamese online handwritten text sampled from pen-based devices. The database stores handwritten text for paragraphs, lines, words, and characters, with the ground truth associated with every paragraph and line. We show a detailed statistical analysis of the handwritten text in this database and describe recognition experiments using several recent methods including the Bidirectional Long Short-Term Memory (BLSTM) network. Overall, our database contains over 480,000 strokes from more than 380,000 characters, which, at present, is the largest database of Vietnamese online handwritten text. Although Vietnamese script is based on a fixed set of alphabet letters, the recognition of Vietnamese online handwritten text poses a difficult challenge because of many diacritical marks, which usually result in delayed strokes during writing. We designed and implemented an online handwriting-collection tool to gather data, as well as a line-segmentation tool and a delayed-stroke-detection tool to analyze collected handwritten text. We also conducted a statistical analysis based on the writer profiles. We applied a number of the state-of-the-art recognition methods on unconstrained Vietnamese handwriting to evaluate their performance, including the BLSTM network, which is an efficient architecture derived from the Recurrent Neural Network (RNN) and is often applied to sequence labeling problems. The BLSTM network achieved 90% character recognition accuracy, despite many long sequences with several delayed strokes. Our database is allowed open access for research to stimulate the development of handwriting research technology.
Clustering Offline Handwritten Mathematical Answers for Computer-Assisted Marking

Vu Tran Minh Khuong, Huy Quang Ung, Cuong Tuan Nguyen, and 1 more author

In Proceedings of the International Conference on Pattern Recognition and Machine Intelligent, ICPRAI 2018, Apr 2018
ICFHR 2018-Competition on Vietnamese online handwritten text recognition using HANDS-VNOnDB (VOHTR2018)

Hung Tuan Nguyen, Cuong Tuan Nguyen, and Masaki Nakagawa

Apr 2018

Abs

This paper presents the results of the VOHTR 2018 competition on Vietnamese Online Handwritten Text Recognition. The goal of this competition is to evaluate and compare recent online handwritten text recognition systems on Vietnamese online handwritten text which contains many delayed strokes caused by the diacritic marks. Besides, the general objective is to encourage the studies on Vietnamese online handwritten text recognition based on the large Vietnamese handwriting database collected from 200 writers. In this competition, we introduce three tasks consisting of word recognition (task 1), text-line recognition (task 2) and paragraph recognition (task 3) which are described in details. Subsequently, we describe the evaluation metrics and give comparative results of competitors along with the brief descriptions of the respective methods.
Online Japanese handwriting recognizers using recurrent neural networks

Hung Tuan Nguyen, Cuong Tuan Nguyen, and Masaki Nakagawa

Apr 2018

Abs

This paper presents an attempt to recognize online isolated handwritten Japanese characters as well as text lines by Recurrent Neural Networks (RNNs). Although there are some successful studies on online Chinese handwriting recognition using RNNs, it is difficult to achieve high accuracy for Japanese due to many different types of characters such as kanji and kana. Moreover, training RNNs usually requires a large number of samples for each class of character. Hence, we apply five different transformation operations on the original samples to generate artificial samples with various deformations. For online handwritten Japanese text, we use the Kondate database, but it does not cover the whole Japanese character set. Thus, we generate random text lines using the sentences from corpora and isolated characters from the Nakayosi and Kuchibue databases. Besides, we employ a robust process with some preprocessing steps and the state-of-the-art online features to extract invariant features from handwritten patterns because the handwritten samples are merged from the different databases collected on different devices with various resolutions. For the recognition model, we have implemented the different Bidirectional Long Short-Term Memory Networks (BLSTM networks) for isolated character classification (sequence classification task) and handwritten text recognition (transcription task). The best model of the sequence classification task achieves an accuracy of 97.91% on Nakayosi and 97.74% on Kuchibue. The best model of transcription task performs a character recognition rate of 86.31% on Kondate and 83.15% on the generated text lines.
Training an end-to-end model for offline handwritten Japanese text recognition by generated synthetic patterns

Nam Tuan Ly, Cuong Tuan Nguyen, and Masaki Nakagawa

In Proceedings of International Conference on Frontiers in Handwriting Recognition, ICFHR 2018, Apr 2018

Abs

This paper presents an end-to-end model of Deep Convolutional Recurrent Network (DCRN) for recognizing offline handwritten Japanese text lines. The end-to-end DCRN model has three parts: A convolutional feature extractor using Deep Convolutional Neural Network (DCNN) to extract a feature sequence from a text line image; recurrent layers employing a Deep Bidirectional LSTM to predict pre-frame from the feature sequence; and a transcription layer using Connectionist Temporal Classification (CTC) to convert the pre-frame predictions into the label sequence. Since our end-to-end model requires a large data for training, we synthesize handwritten text line images from sentences in corpora and handwritten character patterns in the Nakayosi and Kuchibue database with elastic distortions. In the experiment, we evaluate the performance of the end-to-end model and the effectiveness of the synthetic data generation method on the test set of the TUAT Kondate database. The results of the experiments show that our end-to-end model achieves higher than the state-of-the-art recognition accuracy on the test set of TUAT Kondate with 96.35% and 98.05% character level recognition accuracies without and with the generated synthetic data, respectively.

2017

A segmentation method of single- and multiple-touching characters in offline handwritten Japanese text recognition

Kha Cong Nguyen, Cuong Tuan Nguyen, and Masaki Nakagawa

IEICE Transactions on Information and Systems, Apr 2017

Abs

This paper presents a method to segment single- and multiple-touching characters in offline handwritten Japanese text recognition with practical speed. Distortions due to handwriting and a mix of complex Chinese characters with simple phonetic and alphanumeric characters leave optical handwritten text recognition (OHTR) for Japanese still far from perfection. Segmentation of characters, which touch neighbors on multiple points, is a serious unsolved problem. Therefore, we propose a method to segment them which is made in two steps: coarse segmentation and fine segmentation. The coarse segmentation employs vertical projection, stroke-width estimation while the fine segmentation takes a graph-based approach for thinned text images, which employs a new bridge finding process and Voronoi diagrams with two improvements. Unlike previous methods, it locates character centers and seeks segmentation candidates between them. It draws vertical lines explicitly at estimated character centers in order to prevent vertically unconnected components from being left behind in the bridge finding. Multiple candidates of separation are produced by removing touching points combinatorially. SVM is applied to discard improbable segmentation boundaries. Then, ambiguities are finally solved by the text recognition employing linguistic context and geometric context to recognize segmented characters. The results of our experiments show that the proposed method can segment not only single-touching characters but also multiple-touching characters, and each component in our proposed method contributes to the improvement of segmentation and recognition rates.
Attempts to recognize anomalously deformed Kana in Japanese historical documents

Hung Tuan Nguyen, Nam Tuan Ly, Kha Cong Nguyen, and 2 more authors

In Proceedings of the 4th International Workshop on Historical Document Imaging and Processing, HIP 2017, Apr 2017

Abs

This paper presents methods for three different tasks of recognizing anomalously deformed Kana in Japanese historical documents, which were contested by IEICE PRMU1 2017. The tasks have three levels: single character recognition, three Kana characters sequence recognition and unrestricted Kana recognition. We compare several methods for each task. For the level 1, we evaluate CNN based methods and BLSTM based methods. For the level 2, we consider several variations of a combined architecture of CNN and BLSTM. For the level 3, we compare an extension of the method for the level 2 and a segmentation based method. We achieve the single character recognition accuracy of 96.8%, the three Kana characters sequence recognition accuracy of 87.12% and the unrestricted Kana recognition accuracy of 73.3%. These results prove the performance of CNN and BLSTM on these tasks.
Tens of thousands of nom character recognition by deep convolution neural networks

Cong Kha Nguyen, Cuong Tuan Nguyen, and Nakagawa Masaki

In Proceedings of the 4th International Workshop on Historical Document Imaging and Processing, HIP 2017, Apr 2017

Abs

This paper proposes a method to recognize a large set of 32,695 Nom characters which had been used in Vietnam from the tenth century to the twentieth century before the Latin-based Vietnamese alphabet became common. So far, the largest sets to which character recognition methods have been studied, including the latest deep Neural Networks, are about 10,000 for the current set of Chinese, Japanese and Korean, but ancient languages of Chinese origin require much larger sizes of categories. Moreover, lack of training patterns makes the development of Optical Character Recognition (OCR) for Nom be a big challenge. On the other hand, the demand to archive Nom historical documents is very high since a large amount of documents are left uninterpreted and scholars who can comprehend Nom are decreasing. Therefore, we propose a method to recognize a very large set of Nom categories by Deep Convolution Neural Networks (CNN). The proposed method introduces coarse categories which are prepared by K-means beforehand. We construct deep CNNs composed by a coarse category feature extractor, a coarse category classifier and a fine category classifier including some inception modules. We pre-train the coarse category feature extractor and the coarse category classifier with the coarse categories, freeze them and then perform fine tuning to recognize characters in the whole categories of Nom. Unlike conventional cascade of coarse classification and fine classification, the coarse and fine category classifiers are executed in parallel to feature maps generated by the feature extractor and their likelihoods are multiplied. The experiment shows that this architecture provides the better recognition rate than the former cascade of GLVQ and MQDF.