精华内容
下载资源
问答
  • hdfs-site.xml

    2020-12-01 22:57:39
    hdfs-site.xml
  • hdfs-site.xml配置文件详解,有需要的可以下载哈哈哈哈哈
  • HadoopHA高可用集群配置 hdfs-site.xml
  • 修改hdfs-site.xml配置文件

    千次阅读 2019-04-03 21:45:00
    前言 配置Hadoop的主要5个文件 序号 配置文件名 配置对象 主要内容 ...core-site.xml ...用于定义系统级别的参数,如HDFS URL 、Hadoop的临时目录等...hdfs-site.xml HDFS 如名称节点和数据节点的存放位置、文件副本的...

    前言

    配置Hadoop的主要5个文件

    序号配置文件名配置对象主要内容
    1hadoop-env.shhadoop运行环境用来定义hadoop运行环境相关的配置信息
    2core-site.xml集群全局参数用于定义系统级别的参数,如HDFS URL 、Hadoop的临时目录等
    3hdfs-site.xmlHDFS如名称节点和数据节点的存放位置、文件副本的个数、文件的读取权限等
    4mapred-site.xmlMapreduce参数包括JobHistory Server 和应用程序参数两部分,如reduce任务的默认个数、任务所能够使用内存的默认上下限等
    5yarn-site.xml集群资源管理系统参数配置ResourceManager ,nodeManager的通信端口,web监控端口等

    hdfs-site.xml最重要的参数及其解释如下:

    No.参数名默认值参数解释
    1dfs.namenode.secondary.http-address0.0.0.0:50090定义HDFS对应的HTTP服务器地址和端口
    2dfs.namenode.name,dirfile://${hadoop.tmp.dir}/dfs/name定义DFS的名称节点在本地文件系统的位置
    3dfs.datanode.data.dirfile://${hadoop.tmp.dir}/dfs/data定义DFS数据节点存储数据块时存储在本地文件系统的位置
    4dfs.replication3缺省的块复制数量
    5dfs.webhdfs.enabledtrue是否通过http协议读取hdfs文件,如果选是,则集群安全性较差

      本小节修改dfs.replication参数为例
    在这里插入图片描述

    第一步:

      修改前
    在这里插入图片描述
      修改后
    在这里插入图片描述
      dfs.replication是数据需要备份的数量,默认是3,如果此数大于集群的机器数会出错。

    第二步:

      保存退出

    进阶

      通过hadoop官方网址查询core-site.xml 配置信息,可以了解最新的全部的core-site.xml 配置信息,而且包括一些过时的定义标识,从而更好地维护您的集群。

    快速链接

    上一篇汇总目录下一篇
    5.2.2 修改core-site.xml配置文件(二) 云计算数据平台-hadoop集群    5.2.4 修改mapred-site.xml配置文件 (四)
    展开全文
  • * core-site.xml $ ln -s /hadoop/etc/hadoop/core-site.xml /hbase/conf/core-site....$ ln -s /hadoop/etc/hadoop/hdfs-site.xml /hbase/conf/hdfs-site.xml (提示:不要忘记集群的其他几台机器也要做此操作) ...

    * core-site.xml

    $ ln -s /hadoop/etc/hadoop/core-site.xml /hbase/conf/core-site.xml

    * hdfs-site.xml

    $ ln -s /hadoop/etc/hadoop/hdfs-site.xml /hbase/conf/hdfs-site.xml
    (提示:不要忘记集群的其他几台机器也要做此操作)

    展开全文
  • hadoop-hdfs-site.xml配置文件详解

    千次阅读 2020-10-17 17:59:04
    hadoop:hdfs-site.xml配置文件详解 hdfs-site.xml配置文件j介绍 hadoop-hdfs相关的配置信息 hdfs-site.xml配置文件 name value description hadoop.hdfs.configuration.version 1 version of this ...

    hadoop:hdfs-site.xml配置文件详解

    hdfs-site.xml配置文件j介绍

    hadoop-hdfs相关的配置信息

    hdfs-site.xml配置文件

    namevaluedescription
    hadoop.hdfs.configuration.version1version of this configuration file
    dfs.namenode.logging.levelinfoThe logging level for dfs namenode. Other values are “dir” (trace namespace mutations), “block” (trace block under/over replications and block creations/deletions), or “all”.
    dfs.namenode.rpc-addressRPC address that handles all clients requests. In the case of HA/Federation where multiple namenodes exist, the name service id is added to the name e.g. dfs.namenode.rpc-address.ns1 dfs.namenode.rpc-address.EXAMPLENAMESERVICE The value of this property will take the form of nn-host1:rpc-port.
    dfs.namenode.rpc-bind-hostThe actual address the server will bind to. If this optional address is set, the RPC server will bind to this address and the port specified in dfs.namenode.rpc-address for the RPC server. It can also be specified per name node or name service for HA/Federation. This is most useful for making name node listen to all interfaces by setting to 0.0.0.0.
    dfs.namenode.servicerpc-addressRPC address for HDFS Services communication. BackupNode, Datanodes and all other services should be connecting to this address if it is configured. In the case of HA/Federation where multiple namenodes exist, the name service id is added to the name e.g. dfs.namenode.servicerpc-address.ns1 dfs.namenode.rpc-address.EXAMPLENAMESERVICE The value of this property will take the form of nn-host1:rpc-port. If the value of this property is unset the value of dfs.namenode.rpc-address will be used as the default.
    dfs.namenode.servicerpc-bind-hostThe actual address the server will bind to. If this optional address is set, the service RPC server will bind to this address and the port specified in dfs.namenode.servicerpc-address. It can also be specified per name node or name service for HA/Federation. This is most useful for making name node listen to all interfaces by setting to 0.0.0.0.
    dfs.namenode.secondary.http-address0.0.0.0:50090The secondary namenode http server address and port.
    dfs.namenode.secondary.https-address0.0.0.0:50091The secondary namenode HTTPS server address and port.
    dfs.datanode.address0.0.0.0:50010The datanode server address and port for data transfer.
    dfs.datanode.http.address0.0.0.0:50075The datanode http server address and port.
    dfs.datanode.ipc.address0.0.0.0:50020The datanode ipc server address and port. DN的IPC地址和端口。0表示任意空闲端口。 0.0.0.0的含义: 它不只是代表本机,0.0.0.0/8可以表示本网络中的所有主机,0.0.0.0/32可以用作本机的源地址,0.0.0.0/8也可表示本网络上的某个特定主机,综合起来可以说0.0.0.0表示整个网络。它的作用是帮助路由器发送路由表中无法查询的包。如果设置了全零网络的路由,路由表中无法查询的包都将送到全零网络的路由中去。
    dfs.datanode.handler.count10The number of server threads for the datanode. DN的服务线程数。这些线程仅用于接收请求,处理业务命令
    dfs.namenode.http-address0.0.0.0:50070The address and the base port where the dfs namenode web ui will listen on.
    dfs.https.enablefalseDeprecated. Use “dfs.http.policy” instead.
    dfs.http.policyHTTP_ONLYDecide if HTTPS(SSL) is supported on HDFS This configures the HTTP endpoint for HDFS daemons: The following values are supported: - HTTP_ONLY : Service is provided only on http - HTTPS_ONLY : Service is provided only on https - HTTP_AND_HTTPS : Service is provided both on http and https
    dfs.client.https.need-authfalseWhether SSL client certificate authentication is required
    dfs.client.cached.conn.retry3The number of times the HDFS client will pull a socket from the cache. Once this number is exceeded, the client will try to create a new socket.
    dfs.https.server.keystore.resourcessl-server.xmlResource file from which ssl server keystore information will be extracted
    dfs.client.https.keystore.resourcessl-client.xmlResource file from which ssl client keystore information will be extracted
    dfs.datanode.https.address0.0.0.0:50475The datanode secure http server address and port.
    dfs.namenode.https-address0.0.0.0:50470The namenode secure http server address and port.
    dfs.datanode.dns.interfacedefaultThe name of the Network Interface from which a data node should report its IP address.
    dfs.datanode.dns.nameserverdefaultThe host name or IP address of the name server (DNS) which a DataNode should use to determine the host name used by the NameNode for communication and display purposes.
    dfs.namenode.backup.address0.0.0.0:50100The backup node server address and port. If the port is 0 then the server will start on a free port.
    dfs.namenode.backup.http-address0.0.0.0:50105The backup node http server address and port. If the port is 0 then the server will start on a free port.
    dfs.namenode.replication.considerLoadtrueDecide if chooseTarget considers the target’s load or not
    dfs.default.chunk.view.size32768The number of bytes to view for a file on the browser.
    dfs.datanode.du.reserved0Reserved space in bytes per volume. Always leave this much space free for non dfs use.
    dfs.namenode.name.dirfile://${hadoop.tmp.dir}/dfs/nameDetermines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. 本地磁盘目录,NN存储fsimage文件的地方。可以是按逗号分隔的目录列表,fsimage文件会存储在全部目录,冗余安全。这里多个目录设定,最好在多个磁盘,另外,如果其中一个磁盘故障,不会导致系统故障,会跳过坏磁盘。由于使用了HA,建议仅设置一个。如果特别在意安全,可以设置2个
    dfs.namenode.name.dir.restorefalseSet to true to enable NameNode to attempt recovering a previously failed dfs.namenode.name.dir. When enabled, a recovery of any failed directory is attempted during checkpoint.
    dfs.namenode.fs-limits.max-component-length255Defines the maximum number of bytes in UTF-8 encoding in each component of a path. A value of 0 will disable the check.
    dfs.namenode.fs-limits.max-directory-items1048576Defines the maximum number of items that a directory may contain. A value of 0 will disable the check.
    dfs.namenode.fs-limits.min-block-size1048576Minimum block size in bytes, enforced by the Namenode at create time. This prevents the accidental creation of files with tiny block sizes (and thus many blocks), which can degrade performance. 最小的Block大小,字节。在NN创建时强制验证。避免用户设定过小的Block Size,导致过多的Block,这非常影响性能
    dfs.namenode.fs-limits.max-blocks-per-file1048576Maximum number of blocks per file, enforced by the Namenode on write. This prevents the creation of extremely large files which can degrade performance. 每个文件最大的Block数。在NN写时强制检查。用于防止创建超大文件。
    dfs.namenode.edits.dir${dfs.namenode.name.dir}Determines where on the local filesystem the DFS name node should store the transaction (edits) file. If this is a comma-delimited list of directories then the transaction file is replicated in all of the directories, for redundancy. Default value is same as dfs.namenode.name.dir
    dfs.namenode.shared.edits.dirA directory on shared storage between the multiple namenodes in an HA cluster. This directory will be written by the active and read by the standby in order to keep the namespaces synchronized. This directory does not need to be listed in dfs.namenode.edits.dir above. It should be left empty in a non-HA cluster.
    dfs.namenode.edits.journal-plugin.qjournalorg.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager
    dfs.permissions.enabledtrueIf “true”, enable permission checking in HDFS. If “false”, permission checking is turned off, but all other behavior is unchanged. Switching from one parameter value to the other does not change the mode, owner or group of files or directories.
    dfs.permissions.superusergroupsupergroupThe name of the group of super-users.
    dfs.namenode.acls.enabledfalseSet to true to enable support for HDFS ACLs (Access Control Lists). By default, ACLs are disabled. When ACLs are disabled, the NameNode rejects all RPCs related to setting or getting ACLs.
    dfs.block.access.token.enablefalseIf “true”, access tokens are used as capabilities for accessing datanodes. If “false”, no access tokens are checked on accessing datanodes.
    dfs.block.access.key.update.interval600Interval in minutes at which namenode updates its access keys.
    dfs.block.access.token.lifetime600The lifetime of access tokens in minutes.
    dfs.datanode.data.dirfile://${hadoop.tmp.dir}/dfs/dataDetermines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored. 本地磁盘目录,HDFS数据应该存储Block的地方。可以是逗号分隔的目录列表(典型的,每个目录在不同的磁盘)。这些目录被轮流使用,一个块存储在这个目录,下一个块存储在下一个目录,依次循环。每个块在同一个机器上仅存储一份。不存在的目录被忽略。必须创建文件夹,否则被视为不存在。
    dfs.datanode.data.dir.perm700Permissions for the directories on on the local filesystem where the DFS data node store its blocks. The permissions can either be octal or symbolic.
    dfs.replication3Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time. 数据块副本数。此值可以在创建文件是设定,客户端可以只有设定,也可以在命令行修改。不同文件可以有不同的副本数。默认值用于未指定时。
    dfs.replication.max512Maximal block replication. 最大块副本数,不要大于节点总数。
    dfs.namenode.replication.min1Minimal block replication. 最小块副本数。在上传文件时,达到最小副本数,就认为上传是成功的
    dfs.blocksize134217728The default block size for new files, in bytes. You can use the following suffix (case insensitive): k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa) to specify the size (such as 128k, 512m, 1g, etc.), Or provide complete size in bytes (such as 134217728 for 128 MB). 块大小,字节。可以使用后缀: k(kilo), m(mega), g(giga), t(tera), p(peta), e(exa)指定大小 (就像128k, 512m, 1g, 等待)。
    dfs.client.block.write.retries3The number of retries for writing blocks to the data nodes, before we signal failure to the application. 客户端写数据到DN时,最大重试次数。超过重试次数就会报出错误。
    dfs.client.block.write.replace-datanode-on-failure.enabletrueIf there is a datanode/network failure in the write pipeline, DFSClient will try to remove the failed datanode from the pipeline and then continue writing with the remaining datanodes. As a result, the number of datanodes in the pipeline is decreased. The feature is to add new datanodes to the pipeline. This is a site-wide property to enable/disable the feature. When the cluster size is extremely small, e.g. 3 nodes or less, cluster administrators may want to set the policy to NEVER in the default configuration file or disable this feature. Otherwise, users may experience an unusually high rate of pipeline failures since it is impossible to find new datanodes for replacement. See also dfs.client.block.write.replace-datanode-on-failure.policy 在进行pipeline写数据(上传数据的方式)时,如果DN或者磁盘故障,客户端将尝试移除失败的DN,然后写到剩下的磁盘。一个结果是,pipeline中的DN减少了。这个特性是添加新的DN到pipeline。这是一个站点范围的选项。当集群规模非常小时,例如3个或者更小,集群管理者可能想要禁止掉此特性。
    dfs.client.block.write.replace-datanode-on-failure.policyDEFAULTThis property is used only if the value of dfs.client.block.write.replace-datanode-on-failure.enable is true. ALWAYS: always add a new datanode when an existing datanode is removed. NEVER: never add a new datanode. DEFAULT: Let r be the replication number. Let n be the number of existing datanodes. Add a new datanode only if r is greater than or equal to 3 and either (1) floor(r/2) is greater than or equal to n; or (2) r is greater than n and the block is hflushed/appended. 此属性仅在dfs.client.block.write.replace-datanode-on-failure.enable设置为true时有效。 ALWAYS: 总是添加新的DN NEVER: 从不添加新的DN DEFAULT: 设r是副本数,n是要写的DN数。在r>=3并且floor(r/2)>=n或者r>n(前提是文件是hflushed/appended)时添加新的DN。 来自 http://blog.csdn.net/javastart/article/details/50349835
    dfs.blockreport.intervalMsec21600000Determines block reporting interval in milliseconds. 数据块汇报间隔,毫秒,默认是6小时。
    dfs.blockreport.initialDelay0Delay for first block report in seconds. 第一次数据块汇报时延迟,秒。目的是减轻NN压力?
    dfs.blockreport.split.threshold1000000If the number of blocks on the DataNode is below this threshold then it will send block reports for all Storage Directories in a single message. If the number of blocks exceeds this threshold then the DataNode will send block reports for each Storage Directory in separate messages. Set to zero to always split.
    dfs.datanode.directoryscan.interval21600Interval in seconds for Datanode to scan data directories and reconcile the difference between blocks in memory and on the disk. DN的数据块扫描间隔,秒。磁盘上数据和内存中数据调整一致。
    dfs.datanode.directoryscan.threads1How many threads should the threadpool used to compile reports for volumes in parallel have. 线程池要有多少线程用来并发的压缩磁盘的汇报数据。
    dfs.heartbeat.interval3Determines datanode heartbeat interval in seconds. DN的心跳间隔,秒
    dfs.namenode.handler.count10The number of server threads for the namenode. NN的服务线程数。用于处理RPC请求。
    dfs.namenode.safemode.threshold-pct0.999fSpecifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.namenode.replication.min. Values less than or equal to 0 mean not to wait for any particular percentage of blocks before exiting safemode. Values greater than 1 will make safe mode permanent. 数据进入安全模式阀值,百分比,float形式,数据块达到最小副本数(dfs.namenode.replication.min)的百分比。值小于等于0意味着在退出安全模式前不等待数据修复。大于1的值将导致无法离开安全模式。
    dfs.namenode.safemode.min.datanodes0Specifies the number of datanodes that must be considered alive before the name node exits safemode. Values less than or equal to 0 mean not to take the number of live datanodes into account when deciding whether to remain in safe mode during startup. Values greater than the number of datanodes in the cluster will make safe mode permanent. NN收到回报的DN的数量的最小值,达不到此值,NN不退出安全模式。(在系统启动时发生作用)。<=0的值表示不关心DN数量,在启动时。大于DN实际数量的值会导致无法离开安全模式。建议不设置此值
    dfs.namenode.safemode.extension30000Determines extension of safe mode in milliseconds after the threshold level is reached. 安全模式扩展存在时间,在需要的阀值达到后,毫秒。可以设置为0,或者比较短的一个时间,例如3秒
    dfs.datanode.balance.bandwidthPerSec1048576Specifies the maximum amount of bandwidth that each datanode can utilize for the balancing purpose in term of the number of bytes per second. 在做数据平衡时,每个DN最大带宽占用,每秒字节。默认值是1M。建议可以到**10M
    dfs.hostsNames a file that contains a list of hosts that are permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, all hosts are permitted. 文件名,包含了一个host列表,允许列表内机器连到NN。必须指定完整路径。如果值为空,全部hosts都允许连入。
    dfs.hosts.excludeNames a file that contains a list of hosts that are not permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, no hosts are excluded. 文件名,包含了一个hosts列表,不允许列表内机器连到NN。必须指定完整路径。如果值为空。没有host被禁止。如果上述2个都设置并且有重合,dfs.hosts中优先级高*。*
    dfs.namenode.max.objects0The maximum number of files, directories and blocks dfs supports. A value of zero indicates no limit to the number of objects that dfs supports. DFS支持的最大文件、目录、数据块数量。0无限制。
    dfs.namenode.datanode.registration.ip-hostname-checktrueIf true (the default), then the namenode requires that a connecting datanode’s address must be resolved to a hostname. If necessary, a reverse DNS lookup is performed. All attempts to register a datanode from an unresolvable address are rejected. It is recommended that this setting be left on to prevent accidental registration of datanodes listed by hostname in the excludes file during a DNS outage. Only set this to false in environments where there is no infrastructure to support reverse DNS lookup.
    dfs.namenode.decommission.interval30Namenode periodicity in seconds to check if decommission is complete. NN周期性检查退役是否完成的间隔,秒。
    dfs.namenode.decommission.nodes.per.interval5The number of nodes namenode checks if decommission is complete in each dfs.namenode.decommission.interval. NN检查退役是否完成,每dfs.namenode.decommission.interval秒检查的节点数量。
    dfs.namenode.replication.interval3The periodicity in seconds with which the namenode computes repliaction work for datanodes. NN周期性计算DN的副本情况的频率,秒。
    dfs.namenode.accesstime.precision3600000The access time for HDFS file is precise upto this value. The default value is 1 hour. Setting a value of 0 disables access times for HDFS. HDFS文件的访问时间精确到此值,默认是1小时。0表示禁用访问时间。
    dfs.datanode.pluginsComma-separated list of datanode plug-ins to be activated. DN上的插件列表,逗号分隔。
    dfs.namenode.pluginsComma-separated list of namenode plug-ins to be activated. NN上的插件列表,逗号分隔。
    dfs.stream-buffer-size4096The size of buffer to stream files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations. 文件流缓存大小。需要是硬件page大小的整数倍。在读写操作时,数据缓存大小。注意和core-default.xml中指定文件类型的缓存是不同的,这个是dfs共用的
    dfs.bytes-per-checksum512The number of bytes per checksum. Must not be larger than dfs.stream-buffer-size
    dfs.client-write-packet-size65536Packet size for clients to write
    dfs.client.write.exclude.nodes.cache.expiry.interval.millis600000The maximum period to keep a DN in the excluded nodes list at a client. After this period, in milliseconds, the previously excluded node(s) will be removed automatically from the cache and will be considered good for block allocations again. Useful to lower or raise in situations where you keep a file open for very long periods (such as a Write-Ahead-Log (WAL) file) to make the writer tolerant to cluster maintenance restarts. Defaults to 10 minutes.
    dfs.namenode.checkpoint.dirfile://${hadoop.tmp.dir}/dfs/namesecondaryDetermines where on the local filesystem the DFS secondary name node should store the temporary images to merge. If this is a comma-delimited list of directories then the image is replicated in all of the directories for redundancy. 本地文件系统中,DFS SNN应该在哪里存放临时[用于合并|合并后](to merge)的Image。如果是逗号分隔的目录列表,Image文件存放多份。冗余备份。建议不使用SNN功能,忽略此配置
    dfs.namenode.checkpoint.edits.dir${dfs.namenode.checkpoint.dir}Determines where on the local filesystem the DFS secondary name node should store the temporary edits to merge. If this is a comma-delimited list of directoires then teh edits is replicated in all of the directoires for redundancy. Default value is same as dfs.namenode.checkpoint.dir 建议不使用SNN功能,忽略此配置
    dfs.namenode.checkpoint.period3600The number of seconds between two periodic checkpoints.
    dfs.namenode.checkpoint.txns1000000The Secondary NameNode or CheckpointNode will create a checkpoint of the namespace every ‘dfs.namenode.checkpoint.txns’ transactions, regardless of whether ‘dfs.namenode.checkpoint.period’ has expired.
    dfs.namenode.checkpoint.check.period60The SecondaryNameNode and CheckpointNode will poll the NameNode every ‘dfs.namenode.checkpoint.check.period’ seconds to query the number of uncheckpointed transactions.
    dfs.namenode.checkpoint.max-retries3The SecondaryNameNode retries failed checkpointing. If the failure occurs while loading fsimage or replaying edits, the number of retries is limited by this variable.
    dfs.namenode.num.checkpoints.retained2The number of image checkpoint files that will be retained by the NameNode and Secondary NameNode in their storage directories. All edit logs necessary to recover an up-to-date namespace from the oldest retained checkpoint will also be retained.
    dfs.namenode.num.extra.edits.retained1000000The number of extra transactions which should be retained beyond what is minimally necessary for a NN restart. This can be useful for audit purposes or for an HA setup where a remote Standby Node may have been offline for some time and need to have a longer backlog of retained edits in order to start again. Typically each edit is on the order of a few hundred bytes, so the default of 1 million edits should be on the order of hundreds of MBs or low GBs. NOTE: Fewer extra edits may be retained than value specified for this setting if doing so would mean that more segments would be retained than the number configured by dfs.namenode.max.extra.edits.segments.retained. 除最小的必须的editlog之外,额外保留的editlog文件数量。这是有用的,可以用于审核目的,或者HA设置一个远程Standby节点并且有时可能离线时,都需要保留一个较长的backlog。 典型的,每个edit大约几百字节,默认的1百万editlog大约有百兆到1G。注意:早先的extra edits文件可能操作这里设置的值,因为还有其它选项,例如dfs.namenode.max.extra.edits.segments.retained 建议值:2200,约3天的
    dfs.namenode.max.extra.edits.segments.retained10000The maximum number of extra edit log segments which should be retained beyond what is minimally necessary for a NN restart. When used in conjunction with dfs.namenode.num.extra.edits.retained, this configuration property serves to cap the number of extra edits files to a reasonable value.
    dfs.namenode.delegation.key.update-interval86400000The update interval for master key for delegation tokens in the namenode in milliseconds.
    dfs.namenode.delegation.token.max-lifetime604800000The maximum lifetime in milliseconds for which a delegation token is valid.
    dfs.namenode.delegation.token.renew-interval86400000The renewal interval for delegation token in milliseconds.
    dfs.datanode.failed.volumes.tolerated0The number of volumes that are allowed to fail before a datanode stops offering service. By default any volume failure will cause a datanode to shutdown. 可以接受的卷的失败数量。默认值0表示,任一个卷失败都会导致DN关闭。 建议设置此值,避免个别磁盘问题。如果此值超过真实磁盘数,将会报错,启动失败
    dfs.image.compressfalseShould the dfs image be compressed?
    dfs.image.compression.codecorg.apache.hadoop.io.compress.DefaultCodecIf the dfs image is compressed, how should they be compressed? This has to be a codec defined in io.compression.codecs.
    dfs.image.transfer.timeout60000Socket timeout for image transfer in milliseconds. This timeout and the related dfs.image.transfer.bandwidthPerSec parameter should be configured such that normal image transfer can complete successfully. This timeout prevents client hangs when the sender fails during image transfer. This is socket timeout during image tranfer.
    dfs.image.transfer.bandwidthPerSec0Maximum bandwidth used for image transfer in bytes per second. This can help keep normal namenode operations responsive during checkpointing. The maximum bandwidth and timeout in dfs.image.transfer.timeout should be set such that normal image transfers can complete successfully. A default value of 0 indicates that throttling is disabled.
    dfs.image.transfer.chunksize65536Chunksize in bytes to upload the checkpoint. Chunked streaming is used to avoid internal buffering of contents of image file of huge size.
    dfs.namenode.support.allow.formattrueDoes HDFS namenode allow itself to be formatted? You may consider setting this to false for any production cluster, to avoid any possibility of formatting a running DFS. NN是否允许被格式化?在生产系统,把它设置为false,阻止任何格式化操作在一个运行的DFS上。 建议初次格式化后,修改配置禁止
    dfs.datanode.max.transfer.threads4096Specifies the maximum number of threads to use for transferring data in and out of the DN.
    dfs.datanode.readahead.bytes4193404While reading block files, if the Hadoop native libraries are available, the datanode can use the posix_fadvise system call to explicitly page data into the operating system buffer cache ahead of the current reader’s position. This can improve performance especially when disks are highly contended. This configuration specifies the number of bytes ahead of the current read position which the datanode will attempt to read ahead. This feature may be disabled by configuring this property to 0. If the native libraries are not available, this configuration has no effect. 预读磁盘数据。如果Hadoop本地库生效,DN可以调用posix_fadvise系统获取页面数据到操作系统的缓存中。这个配置指定读取当前读取位置之前的字节数。设置为0,取消此功能。无本地库,此功能也无效。
    dfs.datanode.drop.cache.behind.readsfalseIn some workloads, the data read from HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is delivered to the client. This behavior is automatically disabled for workloads which read only short sections of a block (e.g HBase random-IO workloads). This may improve performance for some workloads by freeing buffer cache spage usage for more cacheable data. If the Hadoop native libraries are not available, this configuration has no effect. 在有些场景下,特别是对一些大的,并且不可能重用的数据,缓存在操作系统的缓存区是无用的。此时,DN可以配置自动清理缓存区数据,在已经发生向客户端之后。此功能自动失效,在读取小数据片时。(例如HBase的随机读写场景)。通过释放缓存,这在某些场景下可以提高性能。Hadoop本地库无效,此功能无效。看起来是一个可以尝试的特性
    dfs.datanode.drop.cache.behind.writesfalseIn some workloads, the data written to HDFS is known to be significantly large enough that it is unlikely to be useful to cache it in the operating system buffer cache. In this case, the DataNode may be configured to automatically purge all data from the buffer cache after it is written to disk. This may improve performance for some workloads by freeing buffer cache spage usage for more cacheable data. If the Hadoop native libraries are not available, this configuration has no effect. 同dfs.datanode.drop.cache.behind.reads相似。
    dfs.datanode.sync.behind.writesfalseIf this configuration is enabled, the datanode will instruct the operating system to enqueue all written data to the disk immediately after it is written. This differs from the usual OS policy which may wait for up to 30 seconds before triggering writeback. This may improve performance for some workloads by smoothing the IO profile for data written to disk. If the Hadoop native libraries are not available, this configuration has no effect. 如果是true,在写之后,DN将指示操作系统把队列中数据全部立即写磁盘。和常用的OS策略不同,它们可能在触发写磁盘之前等待30秒。Hadoop本地库无效,此功能无效。
    dfs.client.failover.max.attempts15Expert only. The number of client failover attempts that should be made before the failover is considered failed. 专家设置。客户端失败重试次数。
    dfs.client.failover.sleep.base.millis500Expert only. The time to wait, in milliseconds, between failover attempts increases exponentially as a function of the number of attempts made so far, with a random factor of +/- 50%. This option specifies the base value used in the failover calculation. The first failover will retry immediately. The 2nd failover attempt will delay at least dfs.client.failover.sleep.base.millis milliseconds. And so on.
    dfs.client.failover.sleep.max.millis15000Expert only. The time to wait, in milliseconds, between failover attempts increases exponentially as a function of the number of attempts made so far, with a random factor of +/- 50%. This option specifies the maximum value to wait between failovers. Specifically, the time between two failover attempts will not exceed +/- 50% of dfs.client.failover.sleep.max.millis milliseconds.
    dfs.client.failover.connection.retries0Expert only. Indicates the number of retries a failover IPC client will make to establish a server connection. 专家设置。IPC客户端失败重试次数。在网络不稳定时建议加大此值
    dfs.client.failover.connection.retries.on.timeouts0Expert only. The number of retry attempts a failover IPC client will make on socket timeout when establishing a server connection. 专家设置。IPC客户端失败重试次数,此失败仅指超时失败。在网络不稳定时建议加大此值
    dfs.client.datanode-restart.timeout30Expert only. The time to wait, in seconds, from reception of an datanode shutdown notification for quick restart, until declaring the datanode dead and invoking the normal recovery mechanisms. The notification is sent by a datanode when it is being shutdown using the shutdownDatanode admin command with the upgrade option.
    dfs.nameservicesComma-separated list of nameservices. nameservices列表。逗号分隔。 我们常用的仅配置一个,启动federation功能需要配置多个
    dfs.nameservice.idThe ID of this nameservice. If the nameservice ID is not configured or more than one nameservice is configured for dfs.nameservices it is determined automatically by matching the local node’s address with the configured address.
    dfs.ha.namenodes.EXAMPLENAMESERVICEThe prefix for a given nameservice, contains a comma-separated list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE). 包含一个NN列表。EXAMPLENAMESERVICE是指具体的nameservice名称,通常就是dfs.nameservices中配置的。值是预备配置的NN的ID。 ID是自己取的,不重复就可以,例如nn1,nn2
    dfs.ha.namenode.idThe ID of this namenode. If the namenode ID is not configured it is determined automatically by matching the local node’s address with the configured address.
    dfs.ha.log-roll.period120How often, in seconds, the StandbyNode should ask the active to roll edit logs. Since the StandbyNode only reads from finalized log segments, the StandbyNode will only be as up-to-date as how often the logs are rolled. Note that failover triggers a log roll so the StandbyNode will be up to date before it becomes active. StandbyNode要求Active滚动EditLog,由于StandBy只能从已经完成的Log Segments中读,所以Standby上的数据新鲜程度依赖于以如何的频率滚动日志。秒。另外,故障转移也会触发一次日志滚动,所以StandbyNode在Active之前,数据也会更新成最新的。秒,默认是2分钟。
    dfs.ha.tail-edits.period60How often, in seconds, the StandbyNode should check for new finalized log segments in the shared edits log. StandbyNode以此频率检测共享目录中最新的日志,秒
    dfs.ha.automatic-failover.enabledfalseWhether automatic failover is enabled. See the HDFS High Availability documentation for details on automatic HA configuration. 是否开启自动故障转移。建议开启,true
    dfs.support.appendtrueDoes HDFS allow appends to files?
    dfs.client.use.datanode.hostnamefalseWhether clients should use datanode hostnames when connecting to datanodes.
    dfs.datanode.use.datanode.hostnamefalseWhether datanodes should use datanode hostnames when connecting to other datanodes for data transfer.
    dfs.client.local.interfacesA comma separated list of network interface names to use for data transfer between the client and datanodes. When creating a connection to read from or write to a datanode, the client chooses one of the specified interfaces at random and binds its socket to the IP of that interface. Individual names may be specified as either an interface name (eg “eth0”), a subinterface name (eg “eth0:0”), or an IP address (which may be specified using CIDR notation to match a range of IPs).
    dfs.datanode.shared.file.descriptor.paths/dev/shm,/tmpA comma-separated list of paths to use when creating file descriptors that will be shared between the DataNode and the DFSClient. Typically we use /dev/shm, so that the file descriptors will not be written to disk. Systems that don’t have /dev/shm will fall back to /tmp by default.
    dfs.short.circuit.shared.memory.watcher.interrupt.check.ms60000The length of time in milliseconds that the short-circuit shared memory watcher will go between checking for java interruptions sent from other threads. This is provided mainly for unit tests.
    dfs.namenode.kerberos.internal.spnego.principal${dfs.web.authentication.kerberos.principal}
    dfs.secondary.namenode.kerberos.internal.spnego.principal${dfs.web.authentication.kerberos.principal}
    dfs.namenode.avoid.read.stale.datanodefalseIndicate whether or not to avoid reading from “stale” datanodes whose heartbeat messages have not been received by the namenode for more than a specified time interval. Stale datanodes will be moved to the end of the node list returned for reading. See dfs.namenode.avoid.write.stale.datanode for a similar setting for writes. 决定是否避开从脏DN上读数据。脏DN指在一个指定的时间间隔内没有收到心跳信息。脏DN将被移到可以读取节点列表的尾端。尝试开启
    dfs.namenode.avoid.write.stale.datanodefalseIndicate whether or not to avoid writing to “stale” datanodes whose heartbeat messages have not been received by the namenode for more than a specified time interval. Writes will avoid using stale datanodes unless more than a configured ratio (dfs.namenode.write.stale.datanode.ratio) of datanodes are marked as stale. See dfs.namenode.avoid.read.stale.datanode for a similar setting for reads.
    dfs.namenode.stale.datanode.interval30000Default time interval for marking a datanode as “stale”, i.e., if the namenode has not received heartbeat msg from a datanode for more than this time interval, the datanode will be marked and treated as “stale” by default. The stale interval cannot be too small since otherwise this may cause too frequent change of stale states. We thus set a minimum stale interval value (the default value is 3 times of heartbeat interval) and guarantee that the stale interval cannot be less than the minimum value. A stale data node is avoided during lease/block recovery. It can be conditionally avoided for reads (see dfs.namenode.avoid.read.stale.datanode) and for writes (see dfs.namenode.avoid.write.stale.datanode). 标记一个DN是脏的时间间隔。例如,如果NN在此设定的时间内没有接收到来自某一个节点的心跳信息,此DN将被标记为脏的。此间隔不能太小,否则容易导致被频繁的标记为脏DN。 我们建议是1分钟
    dfs.namenode.write.stale.datanode.ratio0.5fWhen the ratio of number stale datanodes to total datanodes marked is greater than this ratio, stop avoiding writing to stale nodes so as to prevent causing hotspots. 当全部DN被标记为脏DN的比率高于此阀值,停止不写数据到脏DN的策略,以免造成热点问题(有效的,可写的DN太少,压力太大)。
    dfs.namenode.invalidate.work.pct.per.iteration0.32fNote: Advanced property. Change with caution. This determines the percentage amount of block invalidations (deletes) to do over a single DN heartbeat deletion command. The final deletion count is determined by applying this percentage to the number of live nodes in the system. The resultant number is the number of blocks from the deletion list chosen for proper invalidation over a single heartbeat of a single DN. Value should be a positive, non-zero percentage in float notation (X.Yf), with 1.0f meaning 100%.
    dfs.namenode.replication.work.multiplier.per.iteration2Note: Advanced property. Change with caution. This determines the total amount of block transfers to begin in parallel at a DN, for replication, when such a command list is being sent over a DN heartbeat by the NN. The actual number is obtained by multiplying this multiplier with the total number of live nodes in the cluster. The result number is the number of blocks to begin transfers immediately for, per DN heartbeat. This number can be any positive, non-zero integer.
    dfs.nfs.keytab.fileNote: Advanced property. Change with caution. This is the path to the keytab file for the hdfs-nfs gateway. This is required when the cluster is kerberized.
    dfs.nfs.kerberos.principalNote: Advanced property. Change with caution. This is the name of the kerberos principal. This is required when the cluster is kerberized.It must be of this format: nfs-gateway-user/nfs-gateway-host@kerberos-realm
    dfs.webhdfs.enabledtrueEnable WebHDFS (REST API) in Namenodes and Datanodes.
    hadoop.fuse.connection.timeout300The minimum number of seconds that we’ll cache libhdfs connection objects in fuse_dfs. Lower values will result in lower memory consumption; higher values may speed up access by avoiding the overhead of creating new connection objects.
    hadoop.fuse.timer.period5The number of seconds between cache expiry checks in fuse_dfs. Lower values will result in fuse_dfs noticing changes to Kerberos ticket caches more quickly.
    dfs.metrics.percentiles.intervalsComma-delimited set of integers denoting the desired rollover intervals (in seconds) for percentile latency metrics on the Namenode and Datanode. By default, percentile latency metrics are disabled.
    dfs.encrypt.data.transferfalseWhether or not actual block data that is read/written from/to HDFS should be encrypted on the wire. This only needs to be set on the NN and DNs, clients will deduce this automatically. It is possible to override this setting per connection by specifying custom logic via dfs.trustedchannel.resolver.class. 是否加密传输数据?仅需要配置在NN和DN。客户端可以自行判断。
    dfs.encrypt.data.transfer.algorithmThis value may be set to either “3des” or “rc4”. If nothing is set, then the configured JCE default on the system is used (usually 3DES.) It is widely believed that 3DES is more cryptographically secure, but RC4 is substantially faster. 可以设置为"3des"或"rc4"。否则使用默认的,通常是usually 3DES。3DES更安全,RC4更快。
    dfs.trustedchannel.resolver.classTrustedChannelResolver is used to determine whether a channel is trusted for plain data transfer. The TrustedChannelResolver is invoked on both client and server side. If the resolver indicates that the channel is trusted, then the data transfer will not be encrypted even if dfs.encrypt.data.transfer is set to true. The default implementation returns false indicating that the channel is not trusted.
    dfs.datanode.hdfs-blocks-metadata.enabledfalseBoolean which enables backend datanode-side support for the experimental DistributedFileSystem#getFileVBlockStorageLocations API. 布尔值,设定后台DN端是否支持DistributedFileSystem#getFileVBlockStorageLocations API。
    dfs.client.file-block-storage-locations.num-threads10Number of threads used for making parallel RPCs in DistributedFileSystem#getFileBlockStorageLocations(). 在调用DistributedFileSystem#getFileBlockStorageLocations()的并发RPC的线程数
    dfs.client.file-block-storage-locations.timeout.millis1000Timeout (in milliseconds) for the parallel RPCs made in DistributedFileSystem#getFileBlockStorageLocations().
    dfs.journalnode.rpc-address0.0.0.0:8485The JournalNode RPC server address and port.
    dfs.journalnode.http-address0.0.0.0:8480The address and port the JournalNode HTTP server listens on. If the port is 0 then the server will start on a free port.
    dfs.journalnode.https-address0.0.0.0:8481The address and port the JournalNode HTTPS server listens on. If the port is 0 then the server will start on a free port.
    dfs.namenode.audit.loggersdefaultList of classes implementing audit loggers that will receive audit events. These should be implementations of org.apache.hadoop.hdfs.server.namenode.AuditLogger. The special value “default” can be used to reference the default audit logger, which uses the configured log system. Installing custom audit loggers may affect the performance and stability of the NameNode. Refer to the custom logger’s documentation for more details.
    dfs.domain.socket.pathOptional. This is a path to a UNIX domain socket that will be used for communication between the DataNode and local HDFS clients. If the string “_PORT” is present in this path, it will be replaced by the TCP port of the DataNode.
    dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold10737418240Only used when the dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy. This setting controls how much DN volumes are allowed to differ in terms of bytes of free disk space before they are considered imbalanced. If the free space of all the volumes are within this range of each other, the volumes will be considered balanced and block assignments will be done on a pure round robin basis.
    dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction0.75fOnly used when the dfs.datanode.fsdataset.volume.choosing.policy is set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy. This setting controls what percentage of new block allocations will be sent to volumes with more available disk space than others. This setting should be in the range 0.0 - 1.0, though in practice 0.5 - 1.0, since there should be no reason to prefer that volumes with less available disk space receive more block allocations.
    dfs.namenode.edits.noeditlogchannelflushfalseSpecifies whether to flush edit log file channel. When set, expensive FileChannel#force calls are skipped and synchronous disk writes are enabled instead by opening the edit log file with RandomAccessFile(“rws”) flags. This can significantly improve the performance of edit log writes on the Windows platform. Note that the behavior of the “rws” flags is platform and hardware specific and might not provide the same level of guarantees as FileChannel#force. For example, the write will skip the disk-cache on SAS and SCSI devices while it might not on SATA devices. This is an expert level setting, change with caution.
    dfs.client.cache.drop.behind.writesJust like dfs.datanode.drop.cache.behind.writes, this setting causes the page cache to be dropped behind HDFS writes, potentially freeing up more memory for other uses. Unlike dfs.datanode.drop.cache.behind.writes, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. If the native libraries are not available to the DataNode, this configuration has no effect.
    dfs.client.cache.drop.behind.readsJust like dfs.datanode.drop.cache.behind.reads, this setting causes the page cache to be dropped behind HDFS reads, potentially freeing up more memory for other uses. Unlike dfs.datanode.drop.cache.behind.reads, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. If the native libraries are not available to the DataNode, this configuration has no effect.
    dfs.client.cache.readaheadWhen using remote reads, this setting causes the datanode to read ahead in the block file using posix_fadvise, potentially decreasing I/O wait times. Unlike dfs.datanode.readahead.bytes, this is a client-side setting rather than a setting for the entire datanode. If present, this setting will override the DataNode default. When using local reads, this setting determines how much readahead we do in BlockReaderLocal. If the native libraries are not available to the DataNode, this configuration has no effect.
    dfs.namenode.enable.retrycachetrueThis enables the retry cache on the namenode. Namenode tracks for non-idempotent requests the corresponding response. If a client retries the request, the response from the retry cache is sent. Such operations are tagged with annotation @AtMostOnce in namenode protocols. It is recommended that this flag be set to true. Setting it to false, will result in clients getting failure responses to retried request. This flag must be enabled in HA setup for transparent fail-overs. The entries in the cache have expiration time configurable using dfs.namenode.retrycache.expirytime.millis.
    dfs.namenode.retrycache.expirytime.millis600000The time for which retry cache entries are retained.
    dfs.namenode.retrycache.heap.percent0.03fThis parameter configures the heap size allocated for retry cache (excluding the response cached). This corresponds to approximately 4096 entries for every 64MB of namenode process java heap size. Assuming retry cache entry expiration time (configured using dfs.namenode.retrycache.expirytime.millis) of 10 minutes, this enables retry cache to support 7 operations per second sustained for 10 minutes. As the heap size is increased, the operation rate linearly increases.
    dfs.client.mmap.enabledtrueIf this is set to false, the client won’t attempt to perform memory-mapped reads.
    dfs.client.mmap.cache.size256When zero-copy reads are used, the DFSClient keeps a cache of recently used memory mapped regions. This parameter controls the maximum number of entries that we will keep in that cache. The larger this number is, the more file descriptors we will potentially use for memory-mapped files. mmaped files also use virtual address space. You may need to increase your ulimit virtual address space limits before increasing the client mmap cache size. Note that you can still do zero-copy reads when this size is set to 0.
    dfs.client.mmap.cache.timeout.ms3600000The minimum length of time that we will keep an mmap entry in the cache between uses. If an entry is in the cache longer than this, and nobody uses it, it will be removed by a background thread.
    dfs.client.mmap.retry.timeout.ms300000The minimum amount of time that we will wait before retrying a failed mmap operation.
    dfs.client.short.circuit.replica.stale.threshold.ms1800000The maximum amount of time that we will consider a short-circuit replica to be valid, if there is no communication from the DataNode. After this time has elapsed, we will re-fetch the short-circuit replica even if it is in the cache.
    dfs.namenode.path.based.cache.block.map.allocation.percent0.25The percentage of the Java heap which we will allocate to the cached blocks map. The cached blocks map is a hash map which uses chained hashing. Smaller maps may be accessed more slowly if the number of cached blocks is large; larger maps will consume more memory.
    dfs.datanode.max.locked.memory0The amount of memory in bytes to use for caching of block replicas in memory on the datanode. The datanode’s maximum locked memory soft ulimit (RLIMIT_MEMLOCK) must be set to at least this value, else the datanode will abort on startup. By default, this parameter is set to 0, which disables in-memory caching. If the native libraries are not available to the DataNode, this configuration has no effect.
    dfs.namenode.list.cache.directives.num.responses100This value controls the number of cache directives that the NameNode will send over the wire in response to a listDirectives RPC.
    dfs.namenode.list.cache.pools.num.responses100This value controls the number of cache pools that the NameNode will send over the wire in response to a listPools RPC.
    dfs.namenode.path.based.cache.refresh.interval.ms30000The amount of milliseconds between subsequent path cache rescans. Path cache rescans are when we calculate which blocks should be cached, and on what datanodes. By default, this parameter is set to 30 seconds.
    dfs.namenode.path.based.cache.retry.interval.ms30000When the NameNode needs to uncache something that is cached, or cache something that is not cached, it must direct the DataNodes to do so by sending a DNA_CACHE or DNA_UNCACHE command in response to a DataNode heartbeat. This parameter controls how frequently the NameNode will resend these commands.
    dfs.datanode.fsdatasetcache.max.threads.per.volume4The maximum number of threads per volume to use for caching new data on the datanode. These threads consume both I/O and CPU. This can affect normal datanode operations.
    dfs.cachereport.intervalMsec10000Determines cache reporting interval in milliseconds. After this amount of time, the DataNode sends a full report of its cache state to the NameNode. The NameNode uses the cache report to update its map of cached blocks to DataNode locations. This configuration has no effect if in-memory caching has been disabled by setting dfs.datanode.max.locked.memory to 0 (which is the default). If the native libraries are not available to the DataNode, this configuration has no effect.
    dfs.namenode.edit.log.autoroll.multiplier.threshold2.0Determines when an active namenode will roll its own edit log. The actual threshold (in number of edits) is determined by multiplying this value by dfs.namenode.checkpoint.txns. This prevents extremely large edit files from accumulating on the active namenode, which can cause timeouts during namenode startup and pose an administrative hassle. This behavior is intended as a failsafe for when the standby or secondary namenode fail to roll the edit log by the normal checkpoint threshold.
    dfs.namenode.edit.log.autoroll.check.interval.ms300000How often an active namenode will check if it needs to roll its edit log, in milliseconds.
    dfs.webhdfs.user.provider.user.pattern1[A-Za-z0-9._-]*[ ] ? ]? ]?Valid pattern for user and group names for webhdfs, it must be a valid java regex.
    dfs.client.contextdefaultThe name of the DFSClient context that we should use. Clients that share a context share a socket cache and short-circuit cache, among other things. You should only change this if you don’t want to share with another set of threads.
    dfs.namenode.reject-unresolved-dn-topology-mappingfalseIf the value is set to true, then namenode will reject datanode registration if the topology mapping for a datanode is not resolved and NULL is returned (script defined by net.topology.script.file.name fails to execute). Otherwise, datanode will be registered and the default rack will be assigned as the topology path. Topology paths are important for data resiliency, since they define fault domains. Thus it may be unwanted behavior to allow datanode registration with the default rack if the resolving topology failed.

    1. A-Za-z_ ↩︎

    展开全文
  • Hadoop之hdfs-site.xml配置文件

    千次阅读 2019-05-06 16:13:42
    下面来说下hadoop配置hdfs-site.xml文件的配置过程: 增加hdfs配置信息(namenode、datanode端口和目录位置)hdfs-site.xml文件中包含的信息,如复制数据的值,名称节点的路径,本地文件系统的数据节点的路径。 <...

    下面来说下hadoop配置hdfs-site.xml文件的配置过程:

    增加hdfs配置信息(namenode、datanode端口和目录位置)hdfs-site.xml文件中包含的信息,如复制数据的值,名称节点的路径,本地文件系统的数据节点的路径。

    <configuration>
    	<property>
    		<name>dfs.namenode.secondary.http-address</name>
    		<value>bigdata:9001</value>
    	</property>
    	<property>
    		<name>dfs.namenode.name.dir</name>
    		<value>file:/opt/media/hadoop-2.6.0/dfs/name</value>
    	</property>
    	<property>
    		<name>dfs.datanode.data.dir</name>
    		<value>file:/opt/medis/hadoop-2.6.0/dfs/data</value>
    	</property>
    	<property>
    		<name>dfs.replication</name>
    		<value>3</value>
    	</property>
    	<property>
    		<name>dfs.webhdfs.enabled</name>
    		<value>true</value>
    	</property>
    	<property>
    		<name>dfs.permissions</name>
    		<value>false</value>
    	</property>
    	<description>
    		If "true", enable permission checking in HDFS.
    		If "false", permission checking is turned off,but all other behavior is unchanged.
    		Switching from one parameter value to the other does not change the mode, owner or group of files or directories.
    	</description>
    </configuration>
    
    展开全文
  • 配置hadoop,主要是配置core-site.xml,hdfs-site.xml,mapred-site.xml三个配置文件。 上网找的配置可能因为各个hadoop版本不同,导致无法生效,这里需要经验积累。 参数繁多,不用过多纠结每个参数的含义,先照搬...
  • hdfs-site.xml 参数配置

    2019-03-19 09:20:51
    hdfs-site.xml 参数配置 https://www.cnblogs.com/duanxz/p/3799467.html
  • hadoop的伪分布式安装流程...其中core-site.xmlhdfs-site.xml是两个很重要的配置文件。 core-site.xml <configuration> <property> <name>hadoop.tmp.dir</name> <value&...
  • 配置hadoop,主要是配置core-site.xml,hdfs-site.xml,mapred-site.xml三个配置文件,默认下来,这些配置文件都是空的,所以很难知道这些配置文件有哪些配置可以生效,上网找的配置可能因为各个hadoop版本不同,导致...
  • 关于hdfs-site.xml的dfs.replication属性

    千次阅读 2019-08-29 10:20:58
    1.dfs.replication属性的一些概念 dfs.replication是个client参数,即node level参数,需要在每台datanode上设置。默认为3个副本已经够用了,设置...备份系数在hdfs-site.xml中定义,默认值为 如果你只有3个datano...
  • 配置hadoop,主要是配置core-site.xml,hdfs-site.xml,mapred-site.xml三个配置文件,默认下来,这些配置文件都是空的,所以很难知道这些...选择相应版本的hadoop,下载解压后,搜索*.xml,找到core-default.xml,hdfs-...
  • 配置hadoop,主要是配置core-site.xml,hdfs-site.xml,mapred-site.xml三个配置文件,默认下来,这些配置文件都是空的,所以很难知道这些配置文件有哪些配置可以生效,上网找的配置可能因为各个hadoop版本不同,导致...
  • Hadoop的hdfs-site.xml配置描述

    千次阅读 2016-06-03 13:16:10
    hdfs-site.xml的配置项      name  value Description  dfs.default.chunk.view.size 32768 namenode的http访问页面中针对每个文件的内容显示大小,通常无需设置。 dfs....
  • hdfs-site.xml配置项清单: 清单中涉及的专业词语使用如下简称 NN:NameNode DN:DataNode SNN:SecondaryNameNode  属性  值 说明  dfs.namenode.handler.count 10 NN启动后开启的线程数。 dfs....
  • 配置hadoop,主要是配置core-site.xml,hdfs-site.xml,mapred-site.xml三个配置文件,默认下来,这些配置文件都是空的,所以很难知道这些...选择相应版本的hadoop,下载解压后,搜索*.xml,找到core-default.xml,hdfs-...
  • 修改hdfs-site.xml文件的dfs.replication值后,不重启hadoop集群,上传马上生效。 不重启,对于修改dfs.replication值之前的文件备份数不会变动。 重启后,对于修改dfs.replication值之前的文件备份数也不会变动。 ...
  • 配置hadoop,主要是配置core-site.xml,hdfs-site.xml,mapred-site.xml三个配置文件,默认下来,这些配置文件都是空的,所以很难知道这些配置文件有哪些配置可以生效,上网找的配置可能因为各个hadoop版本不同,导致...
  • Hadoop配置 core-site.xml hdfs-site.xml mapred-site.xml 记录一下Hadoop的配置和说明,用到新的配置项会补充进来,不定期更新。以配置文件名划分,以hadoop 1.x配置为例 core-site.xml  name ...
  • 1.hadoop-env.sh 由于Hadoop是java进程,所以需要添加jdk # The java implementation to use. export JAVA_HOME=/usr/local/jdk1.8.0_171/...2.core-site.xml 2.1指定namenode的位置 2.2hadoop.tmp.dir 是hado...
  • [Fatal Error] hdfs-site.xml:22:2: The markup in the document following the root element must be well 表示是hdfs-site.xml配置文件的问题,因为曾经因为为了提交jar包到spark集群,配置了一个忽略用户访问权限...
  • core-site.xml 1)fs.deafulteFS 设置网址(https://.....) 2)ha.zookeeper.qurom 设置高可用(hadoop1:2181,hadoop2:2181,。。。) 3)hadoop.tem.dir 设置临时文件路径(/opt/hadoop)...
  • hadoop-env.sh export JAVA_HOME=/home/hduser/jdk ...core-site.xml <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://master:9000</va...
  • 一 、core-site.xml <!--指定namenode的地址--> <name>fs.defaultFS <value>hdfs://master:9000 <!--用来指定使用hadoop时
  • 一 core-site.xml<configuration><!--指定namenode的地址--> <property> <name>fs.defaultFS</name> <value>hdfs://master:9000</value>...
  • 在搭建hadoop开发环境的时候,需要对hadoop进行配置,其中Hadoop1.x的配置文件放在{hadoop_home}/conf目录下,Hadoop2.x的配置文件放在{hadoop_home}/etc目录下,主要有core-site.xml,hdfs-site.xml,mapred-site.xml...
  • core-site.xml  &lt;property&gt; &lt;name&gt;fs.defaultFS&lt;/name&gt; &lt;value&gt;hdfs://mycluster&lt;/value&gt; &lt;/property&gt; &lt;...

空空如也

空空如也

1 2 3 4 5 ... 20
收藏数 73,200
精华内容 29,280
关键字:

hdfs-site.xml