Proceedings of the 20th Conference on Computer Science and Intelligence Systems (FedCSIS)

DOI: http://dx.doi.org/10.15439/2025F5480

Citation: Proceedings of the 20th Conference on Computer Science and Intelligence Systems (FedCSIS), M. Bolanowski, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 43, pages 291–296 (2025)

Full text

Abstract. Large Language Models (LLMs) capture linguistic structure by operating on sequences of sub-word to- kens, yet they often display behaviors that suggest an implicit grasp of high-level concepts. This study probes whether such ``ideas'' are genuinely encoded in LLM representations and, if so, how faithfully and to what extent. We created a deliberately minimalist LLM (encompassing both tokenizer and transformer archi- tecture)designed to expose internal mechanisms with minimal architectural obscurity. Using a carefully cu- rated toy corpora and probing tasks, we trace how se- mantically related prompts map onto the model's hid- den states. Our findings reveal emergent clustering of conceptually similar inputs even in the stripped-down model. These insights advance our understanding of the representational geometry underpinning modern language models and outline a reproducible framework for future mechanistic studies of semantic abstraction. Keywords: byte-pair encoding, tokenizer, minimalist LLM, sentiment spectrum, idea discovery

References

1. Guilherme Penedo et al. “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale”. In: The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2024. url: https://openreview.net/forum?id=n6SCkn2QaG.
2. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural Machine Translation of Rare Words with Subword Units. 2015. https://dx.doi.org/10.48550/ARXIV.1508.07909. url: https://arxiv.org/abs/1508.07909.
3. Ashish Vaswani et al. Attention Is All You Need. 2017. https://dx.doi.org/10.48550/ARXIV.1706.03762. url: https://arxiv.org/abs/1706.03762.
4. Alec Radford et al. “Language models are unsupervised multitask learners”. In: OpenAI blog 1.8 (2019), p. 9.
5. Tri Dao et al. “Flashattention: Fast and memory-efficient exact attention with io-awareness”. In: Advances in neural information processing systems 35 (2022), pp. 16344–16359.
6. Tom B. Brown et al. Language Models are Few-Shot Learners. 2020. https://dx.doi.org/10.48550/ARXIV.2005.14165. url: https://arxiv.org/abs/2005.14165.
7. Leszek Rolka. “Multi-Criteria Decision-Making by Approximation in the Domain of Linguistic Values”. In: Position Papers of the 18th Conference on Computer Science and Intelligence Systems. Vol. 36. FedCSIS 2023. PTI, Oct. 2023, pp. 97–102. https://dx.doi.org/10.15439/2023f4728.