【Hadoop】Map和Reduce個數問題

在hadoop中當一個任務沒有設置的時候，該任務的執行的map的個數是由任務本身的數據量決定的，具體計算方法會在下文說明；而reduce的個數hadoop是默認設置為1的。為何設置為1那，因為一個任務的輸出的文件個數是由reduce的個數來決定的。一般一個任務的結果默認是輸出到一個文件中，所以reduce的數目設置為1。那如果我們為了提高任務的執行速度如何對map與reduce的個數來進行調整那。

在講解之前首先，看一下hadoop官方文檔是如何說明的。

Number of Maps
The number of maps is usually driven by the number of DFS blocks in the input files. Although that causes people to adjust their DFS block size to adjust the number of maps. The right level of parallelism for maps seems to be around 10-100 maps/node, although we have taken it up to 300 or so for very cpu-light map tasks. Task setup takes awhile, so it is best if the maps take at least a minute to execute.
Actually controlling the number of maps is subtle. The mapred.map.tasks parameter is just a hint to the InputFormat for the number of maps. The default InputFormat behavior is to split the total number of bytes into the right number of fragments. However, in the default case the DFS block size of the input files is treated as an upper bound for input splits. A lower bound on the split size can be set via mapred.min.split.size. Thus, if you expect 10TB of input data and have 128MB DFS blocks, you'll end up with 82k maps, unless your mapred.map.tasks is even larger. Ultimately the InputFormat determines the number of maps.
The number of map tasks can also be increased manually using the JobConf's conf.setNumMapTasks(int num). This can be used to increase the number of map tasks, but will not set the number below that which Hadoop determines via splitting the input data.

Number of Reduces
The right number of reduces seems to be 0.95 or 1.75 * (nodes * mapred.tasktracker.tasks.maximum). At 0.95 all of the reduces can launch immediately and start transfering map outputs as the maps finish. At 1.75 the faster nodes will finish their first round of reduces and launch a second round of reduces doing a much better job of load balancing.
Currently the number of reduces is limited to roughly 1000 by the buffer size for the output files (io.buffer.size * 2 * numReduces << heapSize). This will be fixed at some point, but until it is it provides a pretty firm upper bound.
The number of reduces also controls the number of output files in the output directory, but usually that is not important because the next map/reduce step will split them into even smaller splits for the maps.
The number of reduce tasks can also be increased in the same way as the map tasks, via JobConf's conf.setNumReduceTasks(int num).

上述的說明是map與reduce的個數是如何確定的。對于map的個數是通過任務執行的時候讀入的數據量除以每個block的大小(默認是64M)來決定的，而reduce就是默認為1，而且它有個建議范圍，這個范圍是由你的node個數來決定的。一般reduce的個數是通過：nodes個數 X 一個TaskTracker設置的最大reduce個數(默認為2)  X (0.95~1.75)之間的數目。注意這上述的個數只是設置中的一個最大的上限。在實際運行中的個數，還要看你具體的任務設置。

如果想設置一個任務執行的map與reduce的個數，那可以使用如下方法。

map：當你想更改map的個數的時候，則可以通過更改配置文件中block的size來增大或者減小map的個數，或者通過 JobConf's conf.setNumMapTasks(int num).。但是就算你設置了數目在這里，它在實際運行中的數目不會小于它實際分割產生的數目。意思就是當你通過程序設置map為2個，但是在讀入數據的時候，分割數據是需要3個，那么最后任務在實際運行的過程中map個數是3個而不是你設置的2個。

reduce：當想修改reduce的個數那么可以按照如下方法進行更改：

當是在程序調試中可以通過聲明一個job對象，調用job.setNumReduceTasks(tasks)，或者在conf設置中調用conf.setStrings("mapred.reduce.tasks", values)；

而當是通過命令進行執行任務的時候可以在命令行加入運行期參數：

bin/hadoop jar examples.jar  job_name -Dmapred.map.tasks=nums -Dmapred.reduce.tasks=nums INPUT OUTPUT