Hadoop 验证¶
环境¶
简介¶
Hadoop是一个开源的分布式系统基础架构，核心设计思想是“分而治之”，旨在使用普通的计算机集群处理海量数据。它解决了传统系统无法有效存储与计算互联网时代产生的巨量非结构化和半结构化数据的难题。Hadoop的核心主要由三个组件构成。首先是HDFS（Hadoop Distributed File System），即分布式文件系统。它将超大文件自动切分成多个数据块，并冗余存储在整个集群的多台机器上，提供了高可靠、高吞吐量的数据存储服务。其次是MapReduce，这是一种编程模型，负责分布式并行计算。第三是YARN，作为Hadoop 2.0引入的资源调度平台，它负责集群的资源管理和作业调度，使得Hadoop可以超越MapReduce，运行更多种类的计算框架，如Spark、Flink等，极大地提升了平台的通用性和效率。
安装¶
欧拉镜像修改hostname，免得后面跟hadoop默认配置冲突¶
创建hadoop账户¶
useradd hadoop
chown -R hadoop:hadoop /opt/hadoop-3.4.0
passwd hadoop
su hadoop
nano ~/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.11.9-9.oe2203.riscv64/
进入hadoop目录,做好配置¶
cd /opt/hadoop-3.4.0/
# 配置免密登录ssh
[hadoop@localhost hadoop-3.4.0]$ mkdir data
[hadoop@localhost hadoop-3.4.0]$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
[hadoop@localhost hadoop-3.4.0]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@localhost hadoop-3.4.0]$ chmod 0600 ~/.ssh/authorized_keys
[hadoop@localhost hadoop-3.4.0]$ vi etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop-3.4.0/data/tmp</value> 这里我们指定存放在hadoop根目录下的data文件夹下
</property>
</configuration>
[hadoop@localhost hadoop-3.4.0]$ vi etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
格式化namenode，启动hdfs¶
[hadoop@localhost hadoop-3.4.0]$ bin/hdfs namenode -format
...
2024-06-27 19:49:25,663 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1766065287-127.0.0.1-1719488965584
2024-06-27 19:49:25,670 INFO common.Storage: Will remove files: [/tmp/hadoop-hadoop/dfs/name/current/fsimage_0000000000000000000, /tmp/hadoop-hadoop/dfs/name/current/fsimage_0000000000000000000.md5, /tmp/hadoop-hadoop/dfs/name/current/seen_txid, /tmp/hadoop-hadoop/dfs/name/current/VERSION]
2024-06-27 19:49:25,801 INFO common.Storage: Storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted.
2024-06-27 19:49:26,218 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2024-06-27 19:49:27,452 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 401 bytes saved in 1 seconds .
2024-06-27 19:49:27,588 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2024-06-27 19:49:27,656 INFO blockmanagement.DatanodeManager: Slow peers collection thread shutdown
2024-06-27 19:49:27,841 INFO namenode.FSNamesystem: Stopping services started for active state
2024-06-27 19:49:27,845 INFO namenode.FSNamesystem: Stopping services started for standby state
2024-06-27 19:49:27,896 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2024-06-27 19:49:27,911 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
************************************************************/
[hadoop@localhost hadoop-3.4.0]$ sbin/start-dfs.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [localhost]
检查hdfs是否启动成功¶
[hadoop@localhost hadoop-3.4.0]$ ps -ef|grep java
hadoop 2077 1 99 19:50 ? 00:01:48 /usr/lib/jvm/java-11-openjdk-11.0.11.9-9.oe2203.riscv64//bin/java -Dproc_namenode -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dyarn.log.dir=/opt/hadoop-3.4.0/logs -Dyarn.log.file=hadoop-hadoop-namenode-localhost.log -Dyarn.home.dir=/opt/hadoop-3.4.0 -Dyarn.root.logger=INFO,console -Djava.library.path=/opt/hadoop-3.4.0/lib/native -Dhadoop.log.dir=/opt/hadoop-3.4.0/logs -Dhadoop.log.file=hadoop-hadoop-namenode-localhost.log -Dhadoop.home.dir=/opt/hadoop-3.4.0 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.namenode.NameNode
hadoop 2238 1 99 19:50 ? 00:01:39 /usr/lib/jvm/java-11-openjdk-11.0.11.9-9.oe2203.riscv64//bin/java -Dproc_datanode -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=ERROR,RFAS -Dyarn.log.dir=/opt/hadoop-3.4.0/logs -Dyarn.log.file=hadoop-hadoop-datanode-localhost.log -Dyarn.home.dir=/opt/hadoop-3.4.0 -Dyarn.root.logger=INFO,console -Djava.library.path=/opt/hadoop-3.4.0/lib/native -Dhadoop.log.dir=/opt/hadoop-3.4.0/logs -Dhadoop.log.file=hadoop-hadoop-datanode-localhost.log -Dhadoop.home.dir=/opt/hadoop-3.4.0 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.datanode.DataNode
hadoop 2581 1 99 19:50 ? 00:01:19 /usr/lib/jvm/java-11-openjdk-11.0.11.9-9.oe2203.riscv64//bin/java -Dproc_secondarynamenode -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dyarn.log.dir=/opt/hadoop-3.4.0/logs -Dyarn.log.file=hadoop-hadoop-secondarynamenode-localhost.log -Dyarn.home.dir=/opt/hadoop-3.4.0 -Dyarn.root.logger=INFO,console -Djava.library.path=/opt/hadoop-3.4.0/lib/native -Dhadoop.log.dir=/opt/hadoop-3.4.0/logs -Dhadoop.log.file=hadoop-hadoop-secondarynamenode-localhost.log -Dhadoop.home.dir=/opt/hadoop-3.4.0 -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
hadoop 2933 586 0 19:51 pts/0 00:00:00 grep --color=auto java
[hadoop@localhost hadoop-3.4.0]$ lsof -i:9870
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 2077 hadoop 319u IPv4 11612 0t0 TCP *:9870 (LISTEN)
进行hdfs功能验证¶
[hadoop@localhost hadoop-3.4.0]$ vim /etc/security/limits.conf
//加入以下配置，重启即可生效
* hard nofile 102400
* soft nofile 102400
[hadoop@localhost hadoop-3.4.0]$ bin/hdfs dfs -mkdir -p /user/hadoop
[hadoop@localhost hadoop-3.4.0]$ bin/hdfs dfs -mkdir input
[hadoop@localhost hadoop-3.4.0]$ bin/hdfs dfs -put etc/hadoop/*.xml input/
[hadoop@localhost hadoop-3.4.0]$ bin/hdfs dfs -ls -h input/
Found 10 items
-rw-r--r-- 1 hadoop supergroup 9.0 K 2024-06-27 20:15 input/capacity-scheduler.xml
-rw-r--r-- 1 hadoop supergroup 884 2024-06-27 20:15 input/core-site.xml
-rw-r--r-- 1 hadoop supergroup 13.7 K 2024-06-27 20:15 input/hadoop-policy.xml
-rw-r--r-- 1 hadoop supergroup 683 2024-06-27 20:15 input/hdfs-rbf-site.xml
-rw-r--r-- 1 hadoop supergroup 863 2024-06-27 20:15 input/hdfs-site.xml
-rw-r--r-- 1 hadoop supergroup 620 2024-06-27 20:15 input/httpfs-site.xml
-rw-r--r-- 1 hadoop supergroup 3.4 K 2024-06-27 20:15 input/kms-acls.xml
-rw-r--r-- 1 hadoop supergroup 682 2024-06-27 20:15 input/kms-site.xml
-rw-r--r-- 1 hadoop supergroup 758 2024-06-27 20:15 input/mapred-site.xml
-rw-r--r-- 1 hadoop supergroup 690 2024-06-27 20:15 input/yarn-site.xml
hdfs 删除文件或者目录，目录不为空还不能直接用rmdir删除
[hadoop@55a41a80543f hadoop-3.4.0]$ bin/hdfs dfs -rm -f output/*
Deleted output/_SUCCESS
Deleted output/part-r-00000
[hadoop@55a41a80543f hadoop-3.4.0]$ bin/hdfs dfs -rmdir output
执行mapreduce job测试¶
[hadoop@localhost hadoop-3.4.0]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0.jar grep input output 'dfs[a-z.]+'
2024-06-27 20:17:15,845 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2024-06-27 20:17:17,412 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2024-06-27 20:17:17,426 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2024-06-27 20:17:21,787 INFO input.FileInputFormat: Total input files to process : 10
2024-06-27 20:17:22,111 INFO mapreduce.JobSubmitter: number of splits:10
2024-06-27 20:17:24,429 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local923328979_0001
2024-06-27 20:17:24,431 INFO mapreduce.JobSubmitter: Executing with tokens: []
2024-06-27 20:17:26,121 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2024-06-27 20:17:26,155 INFO mapreduce.Job: Running job: job_local923328979_0001
2024-06-27 20:17:26,170 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2024-06-27 20:17:26,252 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:26,265 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:26,266 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:26,296 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2024-06-27 20:17:26,838 INFO mapred.LocalJobRunner: Waiting for map tasks
2024-06-27 20:17:26,857 INFO mapred.LocalJobRunner: Starting task: attempt_local923328979_0001_m_000000_0
2024-06-27 20:17:27,138 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:27,172 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:27,175 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:27,210 INFO mapreduce.Job: Job job_local923328979_0001 running in uber mode : false
2024-06-27 20:17:27,224 INFO mapreduce.Job: map 0% reduce 0%
2024-06-27 20:17:27,462 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2024-06-27 20:17:27,519 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/hadoop-policy.xml:0+14007
2024-06-27 20:17:28,459 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2024-06-27 20:17:28,461 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2024-06-27 20:17:28,461 INFO mapred.MapTask: soft limit at 83886080
2024-06-27 20:17:28,462 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2024-06-27 20:17:28,462 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2024-06-27 20:17:28,539 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2024-06-27 20:17:29,855 INFO mapred.LocalJobRunner:
2024-06-27 20:17:29,879 INFO mapred.MapTask: Starting flush of map output
2024-06-27 20:17:29,880 INFO mapred.MapTask: Spilling map output
2024-06-27 20:17:29,881 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
2024-06-27 20:17:29,881 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2024-06-27 20:17:30,021 INFO mapred.MapTask: Finished spill 0
2024-06-27 20:17:30,153 INFO mapred.Task: Task:attempt_local923328979_0001_m_000000_0 is done. And is in the process of committing
2024-06-27 20:17:30,210 INFO mapred.LocalJobRunner: map
2024-06-27 20:17:30,212 INFO mapred.Task: Task 'attempt_local923328979_0001_m_000000_0' done.
2024-06-27 20:17:30,287 INFO mapred.Task: Final Counters for attempt_local923328979_0001_m_000000_0: Counters: 24
File System Counters
FILE: Number of bytes read=282212
FILE: Number of bytes written=997555
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=14007
HDFS: Number of bytes written=0
HDFS: Number of read operations=5
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Map input records=325
Map output records=1
Map output bytes=17
Map output materialized bytes=25
Input split bytes=122
Combine input records=1
Combine output records=1
Spilled Records=1
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=208
Total committed heap usage (bytes)=244318208
File Input Format Counters
Bytes Read=14007
2024-06-27 20:17:30,296 INFO mapred.LocalJobRunner: Finishing task: attempt_local923328979_0001_m_000000_0
2024-06-27 20:17:30,308 INFO mapred.LocalJobRunner: Starting task: attempt_local923328979_0001_m_000001_0
2024-06-27 20:17:30,328 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:30,329 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:30,329 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:30,334 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2024-06-27 20:17:30,357 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/capacity-scheduler.xml:0+9213
2024-06-27 20:17:30,503 INFO mapreduce.Job: map 100% reduce 0%
2024-06-27 20:17:30,533 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2024-06-27 20:17:30,535 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2024-06-27 20:17:30,536 INFO mapred.MapTask: soft limit at 83886080
2024-06-27 20:17:30,536 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2024-06-27 20:17:30,536 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2024-06-27 20:17:30,550 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2024-06-27 20:17:30,691 INFO mapred.LocalJobRunner:
2024-06-27 20:17:30,693 INFO mapred.MapTask: Starting flush of map output
2024-06-27 20:17:30,751 INFO mapred.Task: Task:attempt_local923328979_0001_m_000001_0 is done. And is in the process of committing
2024-06-27 20:17:30,794 INFO mapred.LocalJobRunner: map
2024-06-27 20:17:30,795 INFO mapred.Task: Task 'attempt_local923328979_0001_m_000001_0' done.
2024-06-27 20:17:30,805 INFO mapred.Task: Final Counters for attempt_local923328979_0001_m_000001_0: Counters: 24
File System Counters
FILE: Number of bytes read=283438
FILE: Number of bytes written=997593
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=23220
HDFS: Number of bytes written=0
HDFS: Number of read operations=7
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Map input records=244
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=127
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=133
Total committed heap usage (bytes)=244318208
File Input Format Counters
Bytes Read=9213
2024-06-27 20:17:30,806 INFO mapred.LocalJobRunner: Finishing task: attempt_local923328979_0001_m_000001_0
2024-06-27 20:17:30,807 INFO mapred.LocalJobRunner: Starting task: attempt_local923328979_0001_m_000002_0
2024-06-27 20:17:30,819 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:30,820 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:30,820 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:30,824 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2024-06-27 20:17:30,850 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/kms-acls.xml:0+3518
2024-06-27 20:17:30,955 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2024-06-27 20:17:30,955 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2024-06-27 20:17:30,956 INFO mapred.MapTask: soft limit at 83886080
2024-06-27 20:17:30,956 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2024-06-27 20:17:30,956 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2024-06-27 20:17:30,964 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2024-06-27 20:17:31,069 INFO mapred.LocalJobRunner:
2024-06-27 20:17:31,070 INFO mapred.MapTask: Starting flush of map output
2024-06-27 20:17:31,113 INFO mapred.Task: Task:attempt_local923328979_0001_m_000002_0 is done. And is in the process of committing
2024-06-27 20:17:31,146 INFO mapred.LocalJobRunner: map
2024-06-27 20:17:31,147 INFO mapred.Task: Task 'attempt_local923328979_0001_m_000002_0' done.
2024-06-27 20:17:31,156 INFO mapred.Task: Final Counters for attempt_local923328979_0001_m_000002_0: Counters: 24
File System Counters
FILE: Number of bytes read=284664
FILE: Number of bytes written=997631
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=26738
HDFS: Number of bytes written=0
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Map input records=135
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=117
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=59
Total committed heap usage (bytes)=244318208
File Input Format Counters
Bytes Read=3518
2024-06-27 20:17:31,157 INFO mapred.LocalJobRunner: Finishing task: attempt_local923328979_0001_m_000002_0
2024-06-27 20:17:31,158 INFO mapred.LocalJobRunner: Starting task: attempt_local923328979_0001_m_000003_0
2024-06-27 20:17:31,169 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:31,170 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:31,170 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:31,173 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2024-06-27 20:17:31,196 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/core-site.xml:0+884
2024-06-27 20:17:31,302 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2024-06-27 20:17:31,302 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2024-06-27 20:17:31,303 INFO mapred.MapTask: soft limit at 83886080
2024-06-27 20:17:31,303 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2024-06-27 20:17:31,304 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2024-06-27 20:17:31,311 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2024-06-27 20:17:31,445 INFO mapred.LocalJobRunner:
2024-06-27 20:17:31,447 INFO mapred.MapTask: Starting flush of map output
2024-06-27 20:17:31,500 INFO mapred.Task: Task:attempt_local923328979_0001_m_000003_0 is done. And is in the process of committing
2024-06-27 20:17:31,544 INFO mapreduce.Job: map 30% reduce 0%
2024-06-27 20:17:31,548 INFO mapred.LocalJobRunner: map
2024-06-27 20:17:31,549 INFO mapred.Task: Task 'attempt_local923328979_0001_m_000003_0' done.
2024-06-27 20:17:31,555 INFO mapred.Task: Final Counters for attempt_local923328979_0001_m_000003_0: Counters: 24
File System Counters
FILE: Number of bytes read=285890
FILE: Number of bytes written=997669
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=27622
HDFS: Number of bytes written=0
HDFS: Number of read operations=11
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Map input records=24
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=118
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=64
Total committed heap usage (bytes)=299892736
File Input Format Counters
Bytes Read=884
2024-06-27 20:17:31,557 INFO mapred.LocalJobRunner: Finishing task: attempt_local923328979_0001_m_000003_0
2024-06-27 20:17:31,558 INFO mapred.LocalJobRunner: Starting task: attempt_local923328979_0001_m_000004_0
2024-06-27 20:17:31,571 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:31,575 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:31,575 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:31,580 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2024-06-27 20:17:31,601 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/hdfs-site.xml:0+863
2024-06-27 20:17:31,665 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2024-06-27 20:17:31,666 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2024-06-27 20:17:31,667 INFO mapred.MapTask: soft limit at 83886080
2024-06-27 20:17:31,667 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2024-06-27 20:17:31,667 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2024-06-27 20:17:31,694 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2024-06-27 20:17:31,837 INFO mapred.LocalJobRunner:
2024-06-27 20:17:31,839 INFO mapred.MapTask: Starting flush of map output
2024-06-27 20:17:31,839 INFO mapred.MapTask: Spilling map output
2024-06-27 20:17:31,840 INFO mapred.MapTask: bufstart = 0; bufend = 24; bufvoid = 104857600
2024-06-27 20:17:31,842 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2024-06-27 20:17:31,873 INFO mapred.MapTask: Finished spill 0
2024-06-27 20:17:31,913 INFO mapred.Task: Task:attempt_local923328979_0001_m_000004_0 is done. And is in the process of committing
2024-06-27 20:17:31,981 INFO mapred.LocalJobRunner: map
2024-06-27 20:17:31,986 INFO mapred.Task: Task 'attempt_local923328979_0001_m_000004_0' done.
2024-06-27 20:17:31,993 INFO mapred.Task: Final Counters for attempt_local923328979_0001_m_000004_0: Counters: 24
File System Counters
FILE: Number of bytes read=287116
FILE: Number of bytes written=997733
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=28485
HDFS: Number of bytes written=0
HDFS: Number of read operations=13
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Map input records=24
Map output records=1
Map output bytes=24
Map output materialized bytes=32
Input split bytes=118
Combine input records=1
Combine output records=1
Spilled Records=1
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=18
Total committed heap usage (bytes)=299892736
File Input Format Counters
Bytes Read=863
2024-06-27 20:17:31,995 INFO mapred.LocalJobRunner: Finishing task: attempt_local923328979_0001_m_000004_0
2024-06-27 20:17:31,996 INFO mapred.LocalJobRunner: Starting task: attempt_local923328979_0001_m_000005_0
2024-06-27 20:17:32,010 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:32,010 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:32,011 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:32,019 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2024-06-27 20:17:32,050 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/mapred-site.xml:0+758
2024-06-27 20:17:32,115 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2024-06-27 20:17:32,117 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2024-06-27 20:17:32,118 INFO mapred.MapTask: soft limit at 83886080
2024-06-27 20:17:32,119 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2024-06-27 20:17:32,119 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2024-06-27 20:17:32,129 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2024-06-27 20:17:32,289 INFO mapred.LocalJobRunner:
2024-06-27 20:17:32,290 INFO mapred.MapTask: Starting flush of map output
2024-06-27 20:17:32,340 INFO mapred.Task: Task:attempt_local923328979_0001_m_000005_0 is done. And is in the process of committing
2024-06-27 20:17:32,455 INFO mapred.LocalJobRunner: map
2024-06-27 20:17:32,456 INFO mapred.Task: Task 'attempt_local923328979_0001_m_000005_0' done.
2024-06-27 20:17:32,465 INFO mapred.Task: Final Counters for attempt_local923328979_0001_m_000005_0: Counters: 24
File System Counters
FILE: Number of bytes read=287830
FILE: Number of bytes written=997771
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=29243
HDFS: Number of bytes written=0
HDFS: Number of read operations=15
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Map input records=21
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=120
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=15
Total committed heap usage (bytes)=299892736
File Input Format Counters
Bytes Read=758
2024-06-27 20:17:32,466 INFO mapred.LocalJobRunner: Finishing task: attempt_local923328979_0001_m_000005_0
2024-06-27 20:17:32,467 INFO mapred.LocalJobRunner: Starting task: attempt_local923328979_0001_m_000006_0
2024-06-27 20:17:32,478 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:32,479 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:32,479 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:32,484 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2024-06-27 20:17:32,518 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/yarn-site.xml:0+690
2024-06-27 20:17:32,555 INFO mapreduce.Job: map 100% reduce 0%
2024-06-27 20:17:32,579 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2024-06-27 20:17:32,582 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2024-06-27 20:17:32,583 INFO mapred.MapTask: soft limit at 83886080
2024-06-27 20:17:32,584 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2024-06-27 20:17:32,584 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2024-06-27 20:17:32,596 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2024-06-27 20:17:32,719 INFO mapred.LocalJobRunner:
2024-06-27 20:17:32,720 INFO mapred.MapTask: Starting flush of map output
2024-06-27 20:17:32,784 INFO mapred.Task: Task:attempt_local923328979_0001_m_000006_0 is done. And is in the process of committing
2024-06-27 20:17:32,852 INFO mapred.LocalJobRunner: map
2024-06-27 20:17:32,855 INFO mapred.Task: Task 'attempt_local923328979_0001_m_000006_0' done.
2024-06-27 20:17:32,861 INFO mapred.Task: Final Counters for attempt_local923328979_0001_m_000006_0: Counters: 24
File System Counters
FILE: Number of bytes read=288544
FILE: Number of bytes written=997809
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=29933
HDFS: Number of bytes written=0
HDFS: Number of read operations=17
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Map input records=19
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=118
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=18
Total committed heap usage (bytes)=299892736
File Input Format Counters
Bytes Read=690
2024-06-27 20:17:32,862 INFO mapred.LocalJobRunner: Finishing task: attempt_local923328979_0001_m_000006_0
2024-06-27 20:17:32,865 INFO mapred.LocalJobRunner: Starting task: attempt_local923328979_0001_m_000007_0
2024-06-27 20:17:32,902 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:32,902 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:32,903 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:32,908 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2024-06-27 20:17:32,931 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/hdfs-rbf-site.xml:0+683
2024-06-27 20:17:32,994 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2024-06-27 20:17:32,995 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2024-06-27 20:17:32,995 INFO mapred.MapTask: soft limit at 83886080
2024-06-27 20:17:32,995 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2024-06-27 20:17:32,996 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2024-06-27 20:17:33,004 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2024-06-27 20:17:33,112 INFO mapred.LocalJobRunner:
2024-06-27 20:17:33,118 INFO mapred.MapTask: Starting flush of map output
2024-06-27 20:17:33,166 INFO mapred.Task: Task:attempt_local923328979_0001_m_000007_0 is done. And is in the process of committing
2024-06-27 20:17:33,232 INFO mapred.LocalJobRunner: map
2024-06-27 20:17:33,234 INFO mapred.Task: Task 'attempt_local923328979_0001_m_000007_0' done.
2024-06-27 20:17:33,239 INFO mapred.Task: Final Counters for attempt_local923328979_0001_m_000007_0: Counters: 24
File System Counters
FILE: Number of bytes read=289258
FILE: Number of bytes written=997847
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=30616
HDFS: Number of bytes written=0
HDFS: Number of read operations=19
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Map input records=20
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=122
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=21
Total committed heap usage (bytes)=299892736
File Input Format Counters
Bytes Read=683
2024-06-27 20:17:33,240 INFO mapred.LocalJobRunner: Finishing task: attempt_local923328979_0001_m_000007_0
2024-06-27 20:17:33,243 INFO mapred.LocalJobRunner: Starting task: attempt_local923328979_0001_m_000008_0
2024-06-27 20:17:33,280 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:33,281 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:33,282 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:33,298 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2024-06-27 20:17:33,330 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/kms-site.xml:0+682
2024-06-27 20:17:33,403 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2024-06-27 20:17:33,405 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2024-06-27 20:17:33,406 INFO mapred.MapTask: soft limit at 83886080
2024-06-27 20:17:33,406 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2024-06-27 20:17:33,407 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2024-06-27 20:17:33,415 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2024-06-27 20:17:33,525 INFO mapred.LocalJobRunner:
2024-06-27 20:17:33,530 INFO mapred.MapTask: Starting flush of map output
2024-06-27 20:17:33,587 INFO mapreduce.Job: map 80% reduce 0%
2024-06-27 20:17:33,600 INFO mapred.Task: Task:attempt_local923328979_0001_m_000008_0 is done. And is in the process of committing
2024-06-27 20:17:33,643 INFO mapred.LocalJobRunner: map
2024-06-27 20:17:33,644 INFO mapred.Task: Task 'attempt_local923328979_0001_m_000008_0' done.
2024-06-27 20:17:33,651 INFO mapred.Task: Final Counters for attempt_local923328979_0001_m_000008_0: Counters: 24
File System Counters
FILE: Number of bytes read=289972
FILE: Number of bytes written=997885
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=31298
HDFS: Number of bytes written=0
HDFS: Number of read operations=21
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Map input records=20
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=117
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=20
Total committed heap usage (bytes)=299892736
File Input Format Counters
Bytes Read=682
2024-06-27 20:17:33,653 INFO mapred.LocalJobRunner: Finishing task: attempt_local923328979_0001_m_000008_0
2024-06-27 20:17:33,654 INFO mapred.LocalJobRunner: Starting task: attempt_local923328979_0001_m_000009_0
2024-06-27 20:17:33,678 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:33,678 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:33,679 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:33,685 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2024-06-27 20:17:33,729 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/httpfs-site.xml:0+620
2024-06-27 20:17:33,806 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2024-06-27 20:17:33,807 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2024-06-27 20:17:33,808 INFO mapred.MapTask: soft limit at 83886080
2024-06-27 20:17:33,808 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2024-06-27 20:17:33,809 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2024-06-27 20:17:33,824 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2024-06-27 20:17:33,937 INFO mapred.LocalJobRunner:
2024-06-27 20:17:33,938 INFO mapred.MapTask: Starting flush of map output
2024-06-27 20:17:33,990 INFO mapred.Task: Task:attempt_local923328979_0001_m_000009_0 is done. And is in the process of committing
2024-06-27 20:17:34,036 INFO mapred.LocalJobRunner: map
2024-06-27 20:17:34,038 INFO mapred.Task: Task 'attempt_local923328979_0001_m_000009_0' done.
2024-06-27 20:17:34,044 INFO mapred.Task: Final Counters for attempt_local923328979_0001_m_000009_0: Counters: 24
File System Counters
FILE: Number of bytes read=290174
FILE: Number of bytes written=997923
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=31918
HDFS: Number of bytes written=0
HDFS: Number of read operations=23
HDFS: Number of large read operations=0
HDFS: Number of write operations=1
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Map input records=17
Map output records=0
Map output bytes=0
Map output materialized bytes=6
Input split bytes=120
Combine input records=0
Combine output records=0
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=28
Total committed heap usage (bytes)=299892736
File Input Format Counters
Bytes Read=620
2024-06-27 20:17:34,045 INFO mapred.LocalJobRunner: Finishing task: attempt_local923328979_0001_m_000009_0
2024-06-27 20:17:34,048 INFO mapred.LocalJobRunner: map task executor complete.
2024-06-27 20:17:34,087 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2024-06-27 20:17:34,089 INFO mapred.LocalJobRunner: Starting task: attempt_local923328979_0001_r_000000_0
2024-06-27 20:17:34,205 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:34,206 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:34,206 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:34,209 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2024-06-27 20:17:34,239 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7eeaadb3
2024-06-27 20:17:34,255 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2024-06-27 20:17:34,503 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=2212285696, maxSingleShuffleLimit=553071424, mergeThreshold=1460108672, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2024-06-27 20:17:34,527 INFO reduce.EventFetcher: attempt_local923328979_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2024-06-27 20:17:34,595 INFO mapreduce.Job: map 100% reduce 0%
2024-06-27 20:17:34,911 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local923328979_0001_m_000004_0 decomp: 28 len: 32 to MEMORY
2024-06-27 20:17:34,953 INFO reduce.InMemoryMapOutput: Read 28 bytes from map-output for attempt_local923328979_0001_m_000004_0
2024-06-27 20:17:34,990 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 28, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->28
2024-06-27 20:17:35,047 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local923328979_0001_m_000007_0 decomp: 2 len: 6 to MEMORY
2024-06-27 20:17:35,055 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local923328979_0001_m_000007_0
2024-06-27 20:17:35,061 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 28, usedMemory ->30
2024-06-27 20:17:35,092 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local923328979_0001_m_000001_0 decomp: 2 len: 6 to MEMORY
2024-06-27 20:17:35,102 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local923328979_0001_m_000001_0
2024-06-27 20:17:35,108 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 30, usedMemory ->32
2024-06-27 20:17:35,133 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local923328979_0001_m_000000_0 decomp: 21 len: 25 to MEMORY
2024-06-27 20:17:35,141 INFO reduce.InMemoryMapOutput: Read 21 bytes from map-output for attempt_local923328979_0001_m_000000_0
2024-06-27 20:17:35,147 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 21, inMemoryMapOutputs.size() -> 4, commitMemory -> 32, usedMemory ->53
2024-06-27 20:17:35,170 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local923328979_0001_m_000003_0 decomp: 2 len: 6 to MEMORY
2024-06-27 20:17:35,176 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local923328979_0001_m_000003_0
2024-06-27 20:17:35,192 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 53, usedMemory ->55
2024-06-27 20:17:35,216 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local923328979_0001_m_000006_0 decomp: 2 len: 6 to MEMORY
2024-06-27 20:17:35,222 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local923328979_0001_m_000006_0
2024-06-27 20:17:35,224 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 55, usedMemory ->57
2024-06-27 20:17:35,242 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local923328979_0001_m_000009_0 decomp: 2 len: 6 to MEMORY
2024-06-27 20:17:35,250 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local923328979_0001_m_000009_0
2024-06-27 20:17:35,258 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 57, usedMemory ->59
2024-06-27 20:17:35,283 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local923328979_0001_m_000008_0 decomp: 2 len: 6 to MEMORY
2024-06-27 20:17:35,288 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local923328979_0001_m_000008_0
2024-06-27 20:17:35,292 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 59, usedMemory ->61
2024-06-27 20:17:35,322 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local923328979_0001_m_000002_0 decomp: 2 len: 6 to MEMORY
2024-06-27 20:17:35,327 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local923328979_0001_m_000002_0
2024-06-27 20:17:35,332 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 61, usedMemory ->63
2024-06-27 20:17:35,351 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local923328979_0001_m_000005_0 decomp: 2 len: 6 to MEMORY
2024-06-27 20:17:35,356 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local923328979_0001_m_000005_0
2024-06-27 20:17:35,359 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 63, usedMemory ->65
2024-06-27 20:17:35,367 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2024-06-27 20:17:35,380 INFO mapred.LocalJobRunner: 10 / 10 copied.
2024-06-27 20:17:35,382 INFO reduce.MergeManagerImpl: finalMerge called with 10 in-memory map-outputs and 0 on-disk map-outputs
2024-06-27 20:17:35,471 INFO mapred.Merger: Merging 10 sorted segments
2024-06-27 20:17:35,487 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 20 bytes
2024-06-27 20:17:35,512 INFO reduce.MergeManagerImpl: Merged 10 segments, 65 bytes to disk to satisfy reduce memory limit
2024-06-27 20:17:35,521 INFO reduce.MergeManagerImpl: Merging 1 files, 51 bytes from disk
2024-06-27 20:17:35,530 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2024-06-27 20:17:35,531 INFO mapred.Merger: Merging 1 sorted segments
2024-06-27 20:17:35,539 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 29 bytes
2024-06-27 20:17:35,546 INFO mapred.LocalJobRunner: 10 / 10 copied.
2024-06-27 20:17:36,100 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2024-06-27 20:17:36,788 INFO mapred.Task: Task:attempt_local923328979_0001_r_000000_0 is done. And is in the process of committing
2024-06-27 20:17:36,824 INFO mapred.LocalJobRunner: 10 / 10 copied.
2024-06-27 20:17:36,826 INFO mapred.Task: Task attempt_local923328979_0001_r_000000_0 is allowed to commit now
2024-06-27 20:17:37,010 INFO output.FileOutputCommitter: Saved output of task 'attempt_local923328979_0001_r_000000_0' to hdfs://localhost:9000/user/hadoop/grep-temp-207428445
2024-06-27 20:17:37,029 INFO mapred.LocalJobRunner: reduce > reduce
2024-06-27 20:17:37,032 INFO mapred.Task: Task 'attempt_local923328979_0001_r_000000_0' done.
2024-06-27 20:17:37,039 INFO mapred.Task: Final Counters for attempt_local923328979_0001_r_000000_0: Counters: 30
File System Counters
FILE: Number of bytes read=290650
FILE: Number of bytes written=997974
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=31918
HDFS: Number of bytes written=143
HDFS: Number of read operations=28
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=105
Reduce input records=2
Reduce output records=2
Spilled Records=2
Shuffled Maps =10
Failed Shuffles=0
Merged Map outputs=10
GC time elapsed (ms)=0
Total committed heap usage (bytes)=299892736
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Output Format Counters
Bytes Written=143
2024-06-27 20:17:37,040 INFO mapred.LocalJobRunner: Finishing task: attempt_local923328979_0001_r_000000_0
2024-06-27 20:17:37,042 INFO mapred.LocalJobRunner: reduce task executor complete.
2024-06-27 20:17:37,622 INFO mapreduce.Job: map 100% reduce 100%
2024-06-27 20:17:37,629 INFO mapreduce.Job: Job job_local923328979_0001 completed successfully
2024-06-27 20:17:37,959 INFO mapreduce.Job: Counters: 36
File System Counters
FILE: Number of bytes read=3159748
FILE: Number of bytes written=10975390
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=304998
HDFS: Number of bytes written=143
HDFS: Number of read operations=168
HDFS: Number of large read operations=0
HDFS: Number of write operations=13
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Map input records=849
Map output records=2
Map output bytes=41
Map output materialized bytes=105
Input split bytes=1199
Combine input records=2
Combine output records=2
Reduce input groups=2
Reduce shuffle bytes=105
Reduce input records=2
Reduce output records=2
Spilled Records=4
Shuffled Maps =10
Failed Shuffles=0
Merged Map outputs=10
GC time elapsed (ms)=584
Total committed heap usage (bytes)=3132096512
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=31918
File Output Format Counters
Bytes Written=143
2024-06-27 20:17:38,167 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2024-06-27 20:17:38,460 INFO input.FileInputFormat: Total input files to process : 1
2024-06-27 20:17:38,530 INFO mapreduce.JobSubmitter: number of splits:1
2024-06-27 20:17:38,788 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1094844425_0002
2024-06-27 20:17:38,789 INFO mapreduce.JobSubmitter: Executing with tokens: []
2024-06-27 20:17:39,745 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2024-06-27 20:17:39,761 INFO mapreduce.Job: Running job: job_local1094844425_0002
2024-06-27 20:17:39,763 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2024-06-27 20:17:39,769 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:39,773 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:39,773 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:39,783 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2024-06-27 20:17:39,895 INFO mapred.LocalJobRunner: Waiting for map tasks
2024-06-27 20:17:39,897 INFO mapred.LocalJobRunner: Starting task: attempt_local1094844425_0002_m_000000_0
2024-06-27 20:17:39,923 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:39,928 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:39,928 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:39,933 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2024-06-27 20:17:39,956 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/grep-temp-207428445/part-r-00000:0+143
2024-06-27 20:17:40,064 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2024-06-27 20:17:40,065 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2024-06-27 20:17:40,066 INFO mapred.MapTask: soft limit at 83886080
2024-06-27 20:17:40,066 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2024-06-27 20:17:40,066 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2024-06-27 20:17:40,092 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2024-06-27 20:17:40,293 INFO mapred.LocalJobRunner:
2024-06-27 20:17:40,294 INFO mapred.MapTask: Starting flush of map output
2024-06-27 20:17:40,295 INFO mapred.MapTask: Spilling map output
2024-06-27 20:17:40,295 INFO mapred.MapTask: bufstart = 0; bufend = 41; bufvoid = 104857600
2024-06-27 20:17:40,295 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
2024-06-27 20:17:40,317 INFO mapred.MapTask: Finished spill 0
2024-06-27 20:17:40,350 INFO mapred.Task: Task:attempt_local1094844425_0002_m_000000_0 is done. And is in the process of committing
2024-06-27 20:17:40,399 INFO mapred.LocalJobRunner: map
2024-06-27 20:17:40,400 INFO mapred.Task: Task 'attempt_local1094844425_0002_m_000000_0' done.
2024-06-27 20:17:40,404 INFO mapred.Task: Final Counters for attempt_local1094844425_0002_m_000000_0: Counters: 23
File System Counters
FILE: Number of bytes read=571633
FILE: Number of bytes written=1995906
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=32061
HDFS: Number of bytes written=143
HDFS: Number of read operations=34
HDFS: Number of large read operations=0
HDFS: Number of write operations=6
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Map input records=2
Map output records=2
Map output bytes=41
Map output materialized bytes=51
Input split bytes=131
Combine input records=0
Spilled Records=2
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=56
Total committed heap usage (bytes)=299892736
File Input Format Counters
Bytes Read=143
2024-06-27 20:17:40,404 INFO mapred.LocalJobRunner: Finishing task: attempt_local1094844425_0002_m_000000_0
2024-06-27 20:17:40,406 INFO mapred.LocalJobRunner: map task executor complete.
2024-06-27 20:17:40,421 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2024-06-27 20:17:40,424 INFO mapred.LocalJobRunner: Starting task: attempt_local1094844425_0002_r_000000_0
2024-06-27 20:17:40,456 INFO output.PathOutputCommitterFactory: No output committer factory defined, defaulting to FileOutputCommitterFactory
2024-06-27 20:17:40,459 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2024-06-27 20:17:40,459 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2024-06-27 20:17:40,462 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2024-06-27 20:17:40,463 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1d814cae
2024-06-27 20:17:40,470 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2024-06-27 20:17:40,491 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=2212285696, maxSingleShuffleLimit=553071424, mergeThreshold=1460108672, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2024-06-27 20:17:40,498 INFO reduce.EventFetcher: attempt_local1094844425_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2024-06-27 20:17:40,545 INFO reduce.LocalFetcher: localfetcher#2 about to shuffle output of map attempt_local1094844425_0002_m_000000_0 decomp: 47 len: 51 to MEMORY
2024-06-27 20:17:40,567 INFO reduce.InMemoryMapOutput: Read 47 bytes from map-output for attempt_local1094844425_0002_m_000000_0
2024-06-27 20:17:40,580 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 47, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->47
2024-06-27 20:17:40,588 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2024-06-27 20:17:40,594 INFO mapred.LocalJobRunner: 1 / 1 copied.
2024-06-27 20:17:40,595 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2024-06-27 20:17:40,610 INFO mapred.Merger: Merging 1 sorted segments
2024-06-27 20:17:40,615 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 37 bytes
2024-06-27 20:17:40,631 INFO reduce.MergeManagerImpl: Merged 1 segments, 47 bytes to disk to satisfy reduce memory limit
2024-06-27 20:17:40,640 INFO reduce.MergeManagerImpl: Merging 1 files, 51 bytes from disk
2024-06-27 20:17:40,640 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2024-06-27 20:17:40,641 INFO mapred.Merger: Merging 1 sorted segments
2024-06-27 20:17:40,645 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 37 bytes
2024-06-27 20:17:40,647 INFO mapred.LocalJobRunner: 1 / 1 copied.
2024-06-27 20:17:40,766 INFO mapreduce.Job: Job job_local1094844425_0002 running in uber mode : false
2024-06-27 20:17:40,771 INFO mapreduce.Job: map 100% reduce 0%
2024-06-27 20:17:40,869 INFO mapred.Task: Task:attempt_local1094844425_0002_r_000000_0 is done. And is in the process of committing
2024-06-27 20:17:40,902 INFO mapred.LocalJobRunner: 1 / 1 copied.
2024-06-27 20:17:40,903 INFO mapred.Task: Task attempt_local1094844425_0002_r_000000_0 is allowed to commit now
2024-06-27 20:17:40,984 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1094844425_0002_r_000000_0' to hdfs://localhost:9000/user/hadoop/output
2024-06-27 20:17:40,999 INFO mapred.LocalJobRunner: reduce > reduce
2024-06-27 20:17:41,000 INFO mapred.Task: Task 'attempt_local1094844425_0002_r_000000_0' done.
2024-06-27 20:17:41,008 INFO mapred.Task: Final Counters for attempt_local1094844425_0002_r_000000_0: Counters: 30
File System Counters
FILE: Number of bytes read=571767
FILE: Number of bytes written=1995957
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=32061
HDFS: Number of bytes written=172
HDFS: Number of read operations=39
HDFS: Number of large read operations=0
HDFS: Number of write operations=8
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=51
Reduce input records=2
Reduce output records=2
Spilled Records=2
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=0
Total committed heap usage (bytes)=299892736
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Output Format Counters
Bytes Written=29
2024-06-27 20:17:41,009 INFO mapred.LocalJobRunner: Finishing task: attempt_local1094844425_0002_r_000000_0
2024-06-27 20:17:41,014 INFO mapred.LocalJobRunner: reduce task executor complete.
2024-06-27 20:17:41,785 INFO mapreduce.Job: map 100% reduce 100%
2024-06-27 20:17:41,789 INFO mapreduce.Job: Job job_local1094844425_0002 completed successfully
2024-06-27 20:17:41,849 INFO mapreduce.Job: Counters: 36
File System Counters
FILE: Number of bytes read=1143400
FILE: Number of bytes written=3991863
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=64122
HDFS: Number of bytes written=315
HDFS: Number of read operations=73
HDFS: Number of large read operations=0
HDFS: Number of write operations=14
HDFS: Number of bytes read erasure-coded=0
Map-Reduce Framework
Map input records=2
Map output records=2
Map output bytes=41
Map output materialized bytes=51
Input split bytes=131
Combine input records=0
Combine output records=0
Reduce input groups=1
Reduce shuffle bytes=51
Reduce input records=2
Reduce output records=2
Spilled Records=4
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=56
Total committed heap usage (bytes)=599785472
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=143
File Output Format Counters
Bytes Written=29
[hadoop@localhost hadoop-3.4.0]$ bin/hdfs dfs -get output output
[hadoop@localhost hadoop-3.4.0]$ ls output/
part-r-00000 _SUCCESS
[hadoop@localhost hadoop-3.4.0]$ ll output/
total 4.0K
-rw-r--r-- 1 hadoop hadoop 29 Jun 27 20:18 part-r-00000
-rw-r--r-- 1 hadoop hadoop 0 Jun 27 20:18 _SUCCESS
[hadoop@localhost hadoop-3.4.0]$ cat output/part-r-00000
1 dfsadmin
1 dfs.replication
[hadoop@localhost hadoop-3.4.0]$ grep -iR "dfs[a-z.]+" etc/hadoop/*.xml
[hadoop@localhost hadoop-3.4.0]$ grep -iR -E "dfs[a-z.]+" etc/hadoop/*.xml
etc/hadoop/hadoop-policy.xml: dfsadmin and mradmin commands to refresh the security policy in-effect.
etc/hadoop/hdfs-site.xml: <name>dfs.replication</name>