Scalable disk-based topic modeling for memory limited devices
SCIE
SCOPUS
- Title
- Scalable disk-based topic modeling for memory limited devices
- Authors
- BYUNGJU, KIM; DONGHA, LEE; JINOH, OH; HWANJO, YU
- Date Issued
- 2020-04
- Publisher
- ELSEVIER SCIENCE INC
- Abstract
- Disk-based algorithms have the ability to process large-scale data which do not fit into the memory, so they provide good scalability to a mobile device with limited memory resources. In general, the speed of disk I/O is much slower than that of memory access, the total amount of disk I/O is the most crucial factor which determines the efficiency of disk-based algorithms. This paper proposes BlockLDA, an efficient disk-based Latent Dirichlet Allocation (LDA) inference algorithm which can efficiently infer an LDA model when both of the data and model do not fit into the memory. BlockLDA manages the data and model as a set of small blocks so that it can support efficient disk I/O as well as process the LDA inference in a block-wise manner. In addition, it utilizes advanced techniques which help to minimize the amount of disk I/O, including 1) a space reduction algorithm to dynamically manage the block-wise model considering its changing sparsity and 2) a local scheduling algorithm to carefully select the next data blocks so that the number of page faults is minimized. Our experimental results demonstrate that BlockLDA shows better scalability and efficiency than its disk-based and in-memory competitors under the memory-limited environment. (C) 2019 Elsevier Inc. All rights reserved.
- Keywords
- ALGORITHM
- URI
- https://oasis.postech.ac.kr/handle/2014.oak/100696
- DOI
- 10.1016/j.ins.2019.12.058
- ISSN
- 0020-0255
- Article Type
- Article
- Citation
- INFORMATION SCIENCES, vol. 516, page. 353 - 369, 2020-04
- Files in This Item:
- There are no files associated with this item.
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.