A new machine translation method from Japanese start-up Mantra Inc., Yahoo Japan and the University of Tokyo enables global manga fans to enjoy immediate translations of their favourite Japanese comics. The researchers say this is the first comprehensive system for fully automated manga translation from the original Japanese into English or Chinese.
Comics are an integral component of many cultures. US superhero sagas, Korean webtoons and Chinese manhua all have unique styles and characteristics. Most comics however remain readily available only in their domestic markets, due largely to the high cost of translation. To expand the reach of these creative works, the researchers developed an automated system using machine translation (MT) technology.
Manga presents two important problems for MT: context-awareness and multimodal translation. Format-specific challenges include the comics’ unstructured mashup of text and images and the lack of corpora or benchmarks to train and evaluate models.
The team proposes a multimodal context-aware translation framework, which they say is the first to incorporate context information obtained from an image into manga translation. The framework extracts three context types that are useful for multimodal context-aware translation: scene, reading order, and visual information.
The proposed method automatically builds a manga parallel corpus from pairs of original manga texts extracted using computer vision techniques and their translations. The researchers also created a multilingual manga dataset, effectively the first manga translation benchmark, and have made it publicly available.
Given a Japanese manga page as input, the system recognizes Japanese texts and translates them into the target language using the trained neural machine translation model. An image inpainting model meanwhile removes the original Japanese texts from the page, and the translated texts are rendered on the cleaned page with optimized font size and location.
By integrating text recognition, machine translation, and image processing into a unified system, the proposed method achieves impressive automatic image-to-image manga translation and has established a foundation for further research in this area of MT.
The paper Towards Fully Automated Manga Translation is on arXiv.