Logo PTI Logo FedCSIS

Position Papers of the 21st Conference on Computer Science and Intelligence Systems

Annals of Computer Science and Information Systems, Volume 48

Challenges in Stabilizing Training for Diachronic Text Style Transfer

DOI: http://dx.doi.org/10.15439/2026F6104

Citation: Adrian Niedziółka-Domański (). Challenges in Stabilizing Training for Diachronic Text Style Transfer. In M. Bolanowski, M. Ganzha, M. Grzegorowski, L. Maciaszek, M. Paprzycki, A. Paszkiewicz, D. Ślęzak (eds), Proceedings of the 21st Conference on Computer Science and Intelligence Systems. ACSIS, Vol. 48, pages 135–141.

Full text

Abstract. This paper analyzes the challenges associated with training an architecture inspired by CycleGAN for diachronic text style transfer between Early Modern English (EMO) and modern English of the XXI century. Unlike the classic CycleGAN, a single Transformer generator conditioned by the FiLM mecha- nism for the target language period was used, which reduces the number of parameters but significantly complicates the learning process. The main training challenges include the instability of adver- sarial learning, error accumulation in autoregressive generation, loss of semantics, token repetition, and high memory require- ments. This paper proposes solutions that effectively address these issues through a series of mechanisms, such as the use of Gumbel-Softmax, adaptive penalty for token repetition, adaptive control of the discriminator training, or two-stage approach to autoregressive training. Thanks to these mechanisms, it was possible to increase training stability and improve the quality of the generated sequences.

References

  1. J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks,” in 2017 IEEE International Conference on Computer Vision (ICCV). Venice: IEEE, Oct. 2017. https://dx.doi.org/10.1109/ICCV.2017.244. ISBN 978-1-5386-1032-9 pp. 2242–2251. [Online]. Available: http://ieeexplore.ieee.org/document/ 8237506/
  2. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Nets,” in Advances in Neural Information Processing Systems, vol. 27. Curran Associates, Inc., 2014.
  3. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is All you Need,” in Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://proceedings.neurips.cc/paper_files/ paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
  4. E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville, “FiLM: Visual Reasoning with a General Conditioning Layer,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, Apr. 2018. https://dx.doi.org/10.1609/aaai.v32i1.11671. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/11671
  5. A. Niedziółka-Domański and J. Bylina, “Adapting CycleGAN architecture for Unpaired Diachronic Text Style Transfer,” Kraków, Poland, Oct. 2025. https://dx.doi.org/10.15439/2025F4661 pp. 81–86. [Online]. Available: https://annals-csis.org/Volume_44/drp/4661.html
  6. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio, Eds. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019. https://dx.doi.org/10.18653/v1/N19-1423 pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423/
  7. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” in Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc., 2019. [Online]. Available: https://papers.nips.cc/paper_files/paper/2019/ hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html
  8. E. Jang, S. Gu, and B. Poole, “Categorical Reparameterization with Gumbel-Softmax.” arXiv, Aug. 2017. https://dx.doi.org/10.48550/arXiv.1611.01144 ArXiv:1611.01144 [stat.ML]. [Online]. Available: http://arxiv.org/abs/ 1611.01144