Study of Pattern Compression based Big Data Processing System for Large Scale Log Data Analysis 


Vol. 14,  No. 4, pp. 265-272, Apr.  2025
https://doi.org/10.3745/TKIPS.2025.14.4.265


PDF
  Abstract

Distributed systems overcome the limitations of large-scale data processing and address inefficiencies in resource utilization of traditional centralized systems. In particular, large-scale log analysis is utilized in various fields, such as monitoring system performance and operational status and optimizing overall efficiency. Hadoop is a major tool in the field of distributed processing, with MapReduce enabling parallel data processing and HDFS offering high scalability and fault tolerance. These features make Hadoop well-suited for the efficient processing and analysis of large-scale log data. Recently, as IT platforms operate in large-scale distributed environments, the volume of log data has increased exponentially. This rapid growth in log data demands advanced big data analysis technologies but also increases CPU and memory usage, as well as processing delays. To address these challenges, this paper proposes a patterned compression-based big data processing system for large-scale log data analysis. The proposed system leverages data compression technology to reduce resource usage time and consumption while improving processing speed. Performance evaluation results show improvements: CPU usage is reduced by 78.8%, memory usage time by 65%, memory consumption by 6.2%, and processing time by 80%, achieving significant overall performance enhancement. Based on this study, further research and validation are expected to focus on applying patterned compression to various types of big data.

  Statistics
  Cite this article

[IEEE Style]

W. Kim, H. Kim, H. Lee, Y. Kim, "Study of Pattern Compression based Big Data Processing System for Large Scale Log Data Analysis," The Transactions of the Korea Information Processing Society, vol. 14, no. 4, pp. 265-272, 2025. DOI: https://doi.org/10.3745/TKIPS.2025.14.4.265.

[ACM Style]

Wonjib Kim, Hayoon Kim, Hyeopgeon Lee, and Young-Woon Kim. 2025. Study of Pattern Compression based Big Data Processing System for Large Scale Log Data Analysis. The Transactions of the Korea Information Processing Society, 14, 4, (2025), 265-272. DOI: https://doi.org/10.3745/TKIPS.2025.14.4.265.

Forms

Search
(IN TITLE, AUTHOR, ABSTRACT,KEYWORDS)

Advanced Search ▼

POPULAR KEYWORDS
(TOP 10 KEYWORDS)

Recent Publications
(LAST 3 YEARS)

Old Journals

Indexing

All publications of TKIPS is indexed in DOI, EBSCO, Google Scholar, Crossref, and CrossCheck.

TKIPS is also selected as the Journal for Accreditation by NRF (National Research Foundation of Korea).

Related Journals