亚洲激情专区-91九色丨porny丨老师-久久久久久久女国产乱让韩-国产精品午夜小视频观看

溫馨提示×

溫馨提示×

您好,登錄后才能下訂單哦!

密碼登錄×
登錄注冊×
其他方式登錄
點擊 登錄注冊 即表示同意《億速云用戶服務條款》

大數據之---Yarn偽分布式部署和MapReduce案例

發布時間:2020-06-05 18:42:08 來源:網絡 閱讀:1439 作者:ycwyong 欄目:大數據

?

1、軟件環境

RHEL6 角色 jdk-8u45
hadoop-2.8.1.tar.gz ? ssh
xx.xx.xx.xx ip地址 NN hadoop01
xx.xx.xx.xx ip地址 DN hadoop02
xx.xx.xx.xx ip地址 DN hadoop03
xx.xx.xx.xx ip地址 DN hadoop04
xx.xx.xx.xx ip地址 DN hadoop05

本次涉及偽分布式部署只是要主機hadoop01,軟件安裝參考偽分布式部署終極篇

2、配置yarn和mapreduce

?

[hadoop@hadoop000 hadoop]$ cp mapred-site.xml.template mapred-site.xml

配置yarn
[hadoop@hadoop000 hadoop]$ vi mapred-site.xml
<configuration>
? ? <property>
??????? <name>mapreduce.framework.name</name>
??????? <value>yarn</value>
??? </property>

</configuration>

配置mapreduce
[hadoop@hadoop000 hadoop]$ vi yarn-site.xml:
<configuration>
??? <property>
??????? <name>yarn.nodemanager.aux-services</name>
??????? <value>mapreduce_shuffle</value>
??? </property>

</configuration>

?

?

?

3、提交測試jar計算圓周率

job_1524804813835_0001 job命名格式: job_unix時間_數字

大數據之---Yarn偽分布式部署和MapReduce案例

[hadoop@hadoop01 sbin]$ ./start-yarn.sh

[hadoop@hadoop01 hadoop]$ find ./* -name *examples*
./lib/native/examples
./share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.8.1-sources.jar
./share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.8.1-test-sources.jar
./share/hadoop/mapreduce/lib-examples
./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar
./share/doc/hadoop/hadoop-auth-examples
./share/doc/hadoop/hadoop-mapreduce-examples
./share/doc/hadoop/api/org/apache/hadoop/examples
./share/doc/hadoop/api/org/apache/hadoop/security/authentication/examples
[hadoop@hadoop01 hadoop]$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar pi 5 10
Number of Maps? = 5
Samples per Map = 10
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Starting Job
18/04/27 12:58:49 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/04/27 12:58:50 INFO input.FileInputFormat: Total input files to process : 5
18/04/27 12:58:50 INFO mapreduce.JobSubmitter: number of splits:5
18/04/27 12:58:50 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1524804813835_0001
18/04/27 12:58:51 INFO impl.YarnClientImpl: Submitted application application_1524804813835_0001
18/04/27 12:58:51 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1524804813835_0001/
18/04/27 12:58:51 INFO mapreduce.Job: Running job: job_1524804813835_0001
18/04/27 12:59:03 INFO mapreduce.Job: Job job_1524804813835_0001 running in uber mode : false
18/04/27 12:59:03 INFO mapreduce.Job:? map 0% reduce 0%
18/04/27 12:59:18 INFO mapreduce.Job:? map 100% reduce 0%
18/04/27 12:59:25 INFO mapreduce.Job:? map 100% reduce 100%
18/04/27 12:59:26 INFO mapreduce.Job: Job job_1524804813835_0001 completed successfully
18/04/27 12:59:27 INFO mapreduce.Job: Counters: 49
??? File System Counters
??????? FILE: Number of bytes read=116
??????? FILE: Number of bytes written=819783
??????? FILE: Number of read operations=0
??????? FILE: Number of large read operations=0
??????? FILE: Number of write operations=0
??????? HDFS: Number of bytes read=1350
??????? HDFS: Number of bytes written=215
??????? HDFS: Number of read operations=23
??????? HDFS: Number of large read operations=0
??????? HDFS: Number of write operations=3
??? Job Counters
??????? Launched map tasks=5
??????? Launched reduce tasks=1
??????? Data-local map tasks=5
??????? Total time spent by all maps in occupied slots (ms)=64938
??????? Total time spent by all reduces in occupied slots (ms)=4704
??????? Total time spent by all map tasks (ms)=64938
??????? Total time spent by all reduce tasks (ms)=4704
??????? Total vcore-milliseconds taken by all map tasks=64938
??????? Total vcore-milliseconds taken by all reduce tasks=4704
??????? Total megabyte-milliseconds taken by all map tasks=66496512
??????? Total megabyte-milliseconds taken by all reduce tasks=4816896
??? Map-Reduce Framework
??????? Map input records=5
??????? Map output records=10
??????? Map output bytes=90
??????? Map output materialized bytes=140
??????? Input split bytes=760
??????? Combine input records=0
??????? Combine output records=0
??????? Reduce input groups=2
??????? Reduce shuffle bytes=140
??????? Reduce input records=10
??????? Reduce output records=0
??????? Spilled Records=20
??????? Shuffled Maps =5
??????? Failed Shuffles=0
??????? Merged Map outputs=5
??????? GC time elapsed (ms)=1428
??????? CPU time spent (ms)=5740
??????? Physical memory (bytes) snapshot=1536856064
??????? Virtual memory (bytes) snapshot=12578734080
??????? Total committed heap usage (bytes)=1152385024
??? Shuffle Errors
??????? BAD_ID=0
??????? CONNECTION=0
??????? IO_ERROR=0
??????? WRONG_LENGTH=0
??????? WRONG_MAP=0
??????? WRONG_REDUCE=0
??? File Input Format Counters
??????? Bytes Read=590
??? File Output Format Counters
??????? Bytes Written=97
Job Finished in 37.717 seconds
Estimated value of Pi is 3.28000000000000000000
[hadoop@hadoop01 hadoop]$

向AI問一下細節

免責聲明:本站發布的內容(圖片、視頻和文字)以原創、轉載和分享為主,文章觀點不代表本網站立場,如果涉及侵權請聯系站長郵箱:is@yisu.com進行舉報,并提供相關證據,一經查實,將立刻刪除涉嫌侵權內容。

AI

周口市| 京山县| 宣汉县| 宁津县| 兴义市| 门源| 松阳县| 德化县| 金寨县| 青浦区| 交口县| 吉木萨尔县| 张家川| 左权县| 布尔津县| 宁陕县| 罗田县| 三江| 和林格尔县| 石柱| 怀来县| 平原县| 潮安县| 沧源| 天津市| 石城县| 东至县| 沁阳市| 城口县| 北碚区| 梁河县| 金沙县| 张家界市| 锦屏县| 广宁县| 汶川县| 湘潭县| 乌兰察布市| 阿城市| 桐庐县| 和龙市|