hadoop使用内置包进行性能测试TestDFSIO、NNBench、MRBench、SliveTest

hadoop使用内置包进行性能测试

TestDFSIO read & write

TestDFSIO 是一个 Hadoop 自带的基准测试工具，用于测试 HDFS 的读写性能。它会模拟大量的文件读写操作，并输出相应的性能指标。

测试内容：

• TestDFSIO 使用了两个阶段进行测试 — 写入（Write）和读取（Read）。在写入阶段，它将生成指定数量和大小的文件，并将其写入到 HDFS 中。在读取阶段，它将从 HDFS 读取这些文件并计算读取速度。

测试参数：

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java

  private static final Logger LOG = LoggerFactory.getLogger(TestDFSIO.class);
  private static final int DEFAULT_BUFFER_SIZE = 1000000;
  private static final String BASE_FILE_NAME = "test_io_";
  private static final String DEFAULT_RES_FILE_NAME = "TestDFSIO_results.log";
  private static final long MEGA = ByteMultiple.MB.value();
  private static final int DEFAULT_NR_BYTES = 128;
  private static final int DEFAULT_NR_FILES = 4;
  private static final String USAGE =
                    "Usage: " + TestDFSIO.class.getSimpleName() +
                    " [genericOptions]" +
                    " -read [-random | -backward | -skip [-skipSize Size]] |" +
                    " -write | -append | -truncate | -clean" +
                    " [-compression codecClassName]" +
                    " [-nrFiles N]" +
                    " [-size Size[B|KB|MB|GB|TB]]" +
                    " [-resFile resultFileName] [-bufferSize Bytes]" +
                    " [-storagePolicy storagePolicyName]" +
                    " [-erasureCodePolicy erasureCodePolicyName]";

• -write：指定写入阶段的测试。
• -read：指定读取阶段的测试。
• -nrFiles <number>：要创建的文件数量。
• -size <size>：每个文件的大。
• -resFile <filename>：指定结果输出文件的路径。
• -bufferSize <bufferSize>：此参数用于指定读写文件时使用的缓冲区大小。
• compression <codecClassName>：指定在写入文件时使用的压缩编解码器。
• storagePolicy <storagePolicyName>：用于在写入文件时指定存储策略。存储策略决定了数据在 HDFS 中的复制和分布方式。
• erasureCodePolicy <erasureCodePolicyName>：用于在写入文件时指定纠删码（Erasure Code）策略。

执行语句示例：

创建 10 个大小为 100MB 的文件，并将结果输出到当前路径下的TestDFSIO_results.log文件中。

hadoop jar hadoop-mapreduce-client-jobclient-3.3.6-tests.jar TestDFSIO -write -nrFiles 10 -size 100MB

读取10 个大小为 100MB 的文件，并将结果输出到当前路径下的TestDFSIO_results.log文件中。

hadoop jar hadoop-mapreduce-client-test.jar TestDFSIO -read -nrFiles 10 -fileSize 100

TestDFSIO_results.log保存内容：

----- TestDFSIO ----- : write
            Date & time: Thu Dec 14 13:21:57 CST 2023
        Number of files: 10
 Total MBytes processed: 1000
      Throughput mb/sec: 49.72
 Average IO rate mb/sec: 190.11
  IO rate std deviation: 75.6
     Test exec time sec: 27.01

----- TestDFSIO ----- : read
            Date & time: Thu Dec 14 13:36:56 CST 2023
        Number of files: 10
 Total MBytes processed: 1000
      Throughput mb/sec: 631.31
 Average IO rate mb/sec: 655.3
  IO rate std deviation: 127.5
     Test exec time sec: 8.03

NNBench

Hadoop NNbench 是一个用于测试 HDFS （Hadoop 分布式文件系统）性能的工具。它可以模拟多个客户端同时进行读写操作，以评估 HDFS 的并发访问能力和吞吐量。

测试内容：

• 数据生成：NNbench 会生成一系列指定大小的数据块（Blocks），这些数据块将被写入到 HDFS 中。数据块的大小由 -blockSize 参数指定。
• 写入测试：多个客户端同时向 HDFS 写入生成的数据块，可以设置写入的线程数和每个线程的写入次数。写入操作的参数由 -operation create_write 指定。
• 读取测试：多个客户端同时从 HDFS 读取之前写入的数据块。读取操作的参数由 -operation open_read 指定。

测试参数：

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java

Options:
        -operation <Available operations are create_write open_read rename delete. This option is mandatory>
         * NOTE: The open_read, rename and delete operations assume that the files they operate on, are already available. The create_write operation must be run before running the other operations.
        -maps <number of maps. default is 1. This is not mandatory>
        -reduces <number of reduces. default is 1. This is not mandatory>
        -startTime <time to start, given in seconds from the epoch. Make sure this is far enough into the future, so all maps (operations) will start at the same time. default is launch time + 2 mins. This is not mandatory>
        -blockSize <Block size in bytes. default is 1. This is not mandatory>
        -bytesToWrite <Bytes to write. default is 0. This is not mandatory>
        -bytesPerChecksum <Bytes per checksum for the files. default is 1. This is not mandatory>
        -numberOfFiles <number of files to create. default is 1. This is not mandatory>
        -replicationFactorPerFile <Replication factor for the files. default is 1. This is not mandatory>
        -baseDir <base DFS path. default is /benchmarks/NNBench. Supports cross-cluster access by using full path with schema and cluster. This is not mandatory>
        -readFileAfterOpen <true or false. if true, it reads the file and reports the average time to read. This is valid with the open_read operation. default is false. This is not mandatory>
        -help: Display the help statement

• -operation <write|read>：指定进行写入或读取操作。
• -maps <numMaps>：指定 MapReduce 作业中使用的 Mapper 数量。
• -reduces <numReduces>：指定 MapReduce 作业中使用的 Reducer 数量。
• -blockSize <blockSize>：指定生成的数据块的大小。
• -bytesToWrite <bytesToWrite>：指定总共写入的数据大小。
• -bytesPerChecksum <bytesPerChecksum>：指定每个校验和的大小。
• -numberOfFiles <numberOfFiles>：指定创建/读取的文件数。
• -replicationFactorPerFile <replicationFactor>：指定写入数据块的副本数量。
• baseDir <baseDir>：基础DFS路径，默认值为/backages/NNBench。
• readFileAfterOpen <true|false>：如果为true，则读取文件并报告平均读取时间。只对open_read操作有效，默认为false。
• ...

执行语句示例：

写入操作（-opertation create_write），采用 12 个 Mapper 和 6 个 Reducer 来处理数据，每个数据块大小为 64MB（-blockSize 67108864），总共写入 128MB 的数据（-bytesToWrite 134217728），写入文件数为1（-numberOfFiles 1），并且将数据块副本数设置为 2（-replicationFactor 2），基础DFS路径为(/benchmarks/NNBench-’hostname -s’)，测试结果输出到当前路径下的NNBench_results.log文件中。

hadoop jar hadoop-mapreduce-client-jobclient-3.3.6-tests.jar nnbench -operation create_write -maps 12 -reduces 6 -blockSize 67108864 -bytesToWrite 134217728 -numberOfFiles 1 -replicationFactorPerFile 2 -basedir /benchmarks/NNBench-’hostname -s’

读取操作（-opertation open_read），采用 12 个 Mapper 和 6 个 Reducer 来处理数据，每个数据块大小为 64MB（-blockSize 67108864），总共写入 128MB 的数据（-bytesToWrite 134217728），写入文件数为1（-numberOfFiles 1），并且将数据块副本数设置为 2（-replicationFactor 2），报告平均读取时间（-readFileAfterOpen true）基础DFS路径为(/benchmarks/NNBench-’hostname -s’)，测试结果输出到当前路径下的NNBench_results.log文件中。

hadoop jar hadoop-mapreduce-client-jobclient-3.3.6-tests.jar nnbench -operation open_read -maps 12 -reduces 6 -blockSize 67108864 -bytesToWrite 134217728 -numberOfFiles 1 -replicationFactorPerFile 2 -readFileAfterOpen true -basedir /benchmarks/NNBench-’hostname -s’

NNBench_results.log

-------------- NNBench -------------- : 
                               Version: NameNode Benchmark 0.4
                           Date & time: 2023-12-14 14:11:35,409

                        Test Operation: create_write
                            Start time: 2023-12-14 14:11:20,732
                           Maps to run: 12
                        Reduces to run: 6
                    Block Size (bytes): 67108864
                        Bytes to write: 134217728
                    Bytes per checksum: 1
                       Number of files: 1
                    Replication factor: 2
            Successful file operations: 12

        # maps that missed the barrier: 0
                          # exceptions: 0

               TPS: Create/Write/Close: 2
Avg exec time (ms): Create/Write/Close: 882.25
            Avg Lat (ms): Create/Write: 670.9166666666666
                   Avg Lat (ms): Close: 109.58333333333333

                 RAW DATA: AL Total #1: 8051
                 RAW DATA: AL Total #2: 1315
              RAW DATA: TPS Total (ms): 10587
       RAW DATA: Longest Map Time (ms): 11954.0
                   RAW DATA: Late maps: 0
             RAW DATA: # of exceptions: 0

-------------- NNBench -------------- : 
                               Version: NameNode Benchmark 0.4
                           Date & time: 2023-12-14 14:17:34,927

                        Test Operation: open_read
                            Start time: 2023-12-14 14:17:26,840
                           Maps to run: 12
                        Reduces to run: 6
                    Block Size (bytes): 67108864
                        Bytes to write: 134217728
                    Bytes per checksum: 1
                       Number of files: 1
                    Replication factor: 2
            Successful file operations: 12

        # maps that missed the barrier: 0
                          # exceptions: 0

                        TPS: Open/Read: 2
         Avg Exec time (ms): Open/Read: 320.75
                    Avg Lat (ms): Open: 3.0
                  Avg Lat (ms): Read: 175.75

                 RAW DATA: AL Total #1: 36
                 RAW DATA: AL Total #2: 2109
              RAW DATA: TPS Total (ms): 3849
       RAW DATA: Longest Map Time (ms): 4837.0
                   RAW DATA: Late maps: 0
             RAW DATA: # of exceptions: 0

MRBench

Hadoop Mrbench 是一个用于测试 Hadoop MapReduce 的性能的工具。它可以模拟 MapReduce 作业的执行过程，以评估 Hadoop 集群的并发处理能力和吞吐量。

测试内容：

• 数据生成：Mrbench 会生成指定数量的输入数据块（blocks），每个数据块的大小由 -blockSize 参数指定。
• MapReduce 作业执行：通过调度 MapReduce 作业来模拟实际的计算任务。作业的类型、数量和规模由 -numRuns、-maps 和 -reduces 参数指定。
• 数据读取：测试过程中，将对生成的数据块进行读取操作，以验证数据的正确性。

测试参数：

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MRBench.java

Usage: mrbench [-baseDir <base DFS path for output/input, default is /benchmarks/MRBench>] [-jar <local path to job jar file containing Mapper and Reducer implementations, default is current jar file>] [-numRuns <number of times to run the job, default is 1>] [-maps <number of maps for each run, default is 2>] [-reduces <number of reduces for each run, default is 1>] [-inputLines <number of input lines to generate, default is 1>] [-inputType <type of input to generate, one of ascending (default), descending, random>] [-verbose]

• -maps <numMaps>：指定 MapReduce 作业中使用的 Mapper 数量。
• -reduces <numReduces>：指定 MapReduce 作业中使用的 Reducer 数量。
• -numRuns <numRuns>：指定要执行的 MapReduce 作业的次数。
• -inputLines：指定要输入的数据行数
• ...

模拟运行10次 MapReduce 作业（-numRuns 10），使用 12 个 Mapper 和 6 个 Reducer 来处理数据（-maps 12 -reduces 6），每个文件输入数据行数为50。
执行语句示例：

hadoop jar hadoop-mapreduce-client-jobclient-3.3.6-tests.jar mrbench -maps 12 -reduces 6 -numRuns 10 -inputLines 50

DataLines       Maps    Reduces AvgTime (milliseconds)
50              12      6       2683

SliveTest

SliveTest 测试的主要目的是评估 Hadoop 文件系统（HDFS）和 MapReduce 框架在高负载、并发访问以及数据完整性方面的表现。
由于 SliveTest 类是作为 Hadoop MapReduce 项目中的一个测试类存在的，它通常不作为独立工具来使用，而是通过运行整个测试套件进行测试。

测试参数：

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/SliveTest.java

usage: SliveTest 0.1.0
 -append <arg>         pct,distribution where distribution is one of
                       beg,end,uniform,mid
 -appendSize <arg>     Min,max for size to append
                       (min=max=MAX_LONG=blocksize)
 -baseDir <arg>        Base directory path
 -blockSize <arg>      Min,max for dfs file block size
 -cleanup <arg>        Cleanup & remove directory after reporting
 -create <arg>         pct,distribution where distribution is one of
                       beg,end,uniform,mid
 -delete <arg>         pct,distribution where distribution is one of
                       beg,end,uniform,mid
 -dirSize <arg>        Max files per directory
 -duration <arg>       Duration of a map task in seconds (MAX_INT for no
                       limit)
 -exitOnError          Exit on first error
 -files <arg>          Max total number of files
 -help                 Usage information
 -ls <arg>             pct,distribution where distribution is one of
                       beg,end,uniform,mid
 -maps <arg>           Number of maps
 -mkdir <arg>          pct,distribution where distribution is one of
                       beg,end,uniform,mid
 -ops <arg>            Max number of operations per map
 -packetSize <arg>     Dfs write packet size
 -queue <arg>          Queue name
 -read <arg>           pct,distribution where distribution is one of
                       beg,end,uniform,mid
 -readSize <arg>       Min,max for size to read (min=max=MAX_LONG=read
                       entire file)
 -reduces <arg>        Number of reduces
 -rename <arg>         pct,distribution where distribution is one of
                       beg,end,uniform,mid
 -replication <arg>    Min,max value for replication amount
 -resFile <arg>        Result file name
 -seed <arg>           Random number seed
 -sleep <arg>          Min,max for millisecond of random sleep to perform
                       (between operations)
 -truncate <arg>       pct,distribution where distribution is one of
                       beg,end,uniform,mid
 -truncateSize <arg>   Min,max for size to truncate
                       (min=max=MAX_LONG=blocksize)
 -truncateWait <arg>   Should wait for truncate recovery
 -writeSize <arg>      Min,max for size to write
                       (min=max=MAX_LONG=blocksize)

• -append <arg>：追加操作的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。
• -appendSize <arg>：追加操作的最小和最大文件大小。如果未指定，则默认为 MAX_LONG=blocksize。
• -baseDir <arg>：基础目录路径，即执行操作的根目录。
• -blockSize <arg>：HDFS 文件块大小的最小和最大值。
• -cleanup <arg>：在报告后进行清理并删除目录。<arg> 可以是 true 或 false。
• -create <arg>：创建文件的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。
• -delete <arg>：删除操作的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。
• -dirSize <arg>：每个目录的最大文件数。
• -duration <arg>：每个 Map 任务的持续时间（秒）。设置为 MAX_INT 表示没有时间限制。
• -exitOnError：在出现错误时立即退出。
• -files <arg>：总文件数的最大值。
• -help：显示使用信息。
• -ls <arg>：列出操作的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。
• -maps <arg>：Map 任务的数量。
• -mkdir <arg>：创建目录的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。
• -ops <arg>：每个 Map 任务的最大操作数。
• -packetSize <arg>：DFS 写入数据包的大小。
• -queue <arg>：指定作业提交到的队列名称。
• -read <arg>：读取操作的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。
• -readSize <arg>：读取操作的最小和最大文件大小。如果未指定，则默认为 MAX_LONG=read entire file。
• -reduces <arg>：Reduce 任务的数量。
• -rename <arg>：重命名操作的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。
• -replication <arg>：文件的最小和最大副本数量。
• -resFile <arg>：结果文件名。
• -seed <arg>：随机数种子。
• -sleep <arg>：执行操作之间的随机休眠时间，单位为毫秒。指定一个最小值和最大值。
• -truncate <arg>：截断操作的百分比和分布方式。分布方式可以是以下之一：beg（起始）、end（结束）、uniform（均匀）、mid（中间）。
• -truncateSize <arg>：截断操作的最小和最大文件大小。如果未指定，则默认为 MIN_LONG=blocksize。
• -truncateWait <arg>：是否等待截断恢复完成。<arg> 可以是 true 或 false。
• -writeSize <arg>: …

hadoop org.apache.hadoop.fs.slive.SliveTest -maps 12 -reduces 6 -files 5 -ops 5 -truncateWait true -cleanup true

执行完毕后如果没有设置 -cleanup true 则需要手动清理测试痕迹：

hdfs dfs -rm -r -f hdfs://hadoop-system-namenode:9000/test/slive/slive/output

2023-12-14 16:51:05,863 INFO slive.SliveTest: Reporting on job:
2023-12-14 16:51:05,863 INFO slive.SliveTest: Writing report using contents of /test/slive/slive/output
2023-12-14 16:51:05,925 INFO slive.SliveTest: Report results being placed to logging output and to file /opt/hadoop/share/hadoop/mapreduce/part-0000
2023-12-14 16:51:05,926 INFO slive.ReportWriter: Basic report for operation type AppendOp
2023-12-14 16:51:05,926 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,926 INFO slive.ReportWriter: Measurement "files_not_found" = 1
2023-12-14 16:51:05,926 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Basic report for operation type CreateOp
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "bytes_written" = 67108864
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "milliseconds_taken" = 3018
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "successes" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Rate for measurement "bytes_written" = 21.206 MB/sec
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Rate for measurement "op_count" = 0.331 operations/sec
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Rate for measurement "successes" = 0.331 successes/sec
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Basic report for operation type DeleteOp
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "failures" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Basic report for operation type ListOp
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "files_not_found" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Basic report for operation type MkdirOp
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "milliseconds_taken" = 35
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "successes" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Rate for measurement "op_count" = 28.571 operations/sec
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Rate for measurement "successes" = 28.571 successes/sec
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Basic report for operation type ReadOp
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "files_not_found" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Basic report for operation type RenameOp
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "failures" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Basic report for operation type SliveMapper
2023-12-14 16:51:05,927 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,927 INFO slive.ReportWriter: Measurement "milliseconds_taken" = 4486
2023-12-14 16:51:05,928 INFO slive.ReportWriter: Measurement "op_count" = 8
2023-12-14 16:51:05,928 INFO slive.ReportWriter: Rate for measurement "op_count" = 1.783 operations/sec
2023-12-14 16:51:05,928 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,928 INFO slive.ReportWriter: Basic report for operation type TruncateOp
2023-12-14 16:51:05,928 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,928 INFO slive.ReportWriter: Measurement "files_not_found" = 1
2023-12-14 16:51:05,928 INFO slive.ReportWriter: Measurement "op_count" = 1
2023-12-14 16:51:05,928 INFO slive.ReportWriter: -------------
2023-12-14 16:51:05,928 INFO slive.SliveTest: Cleaning up job:
2023-12-14 16:51:05,928 INFO slive.SliveTest: Attempting to recursively delete /test/slive/slive