您当前位置：首页 > 服务器 > Hadoop的distcp命令出现Permission denied错误

Hadoop的distcp命令出现Permission denied错误

来源：程序员人生发布时间：2015-06-08 08:25:51 阅读次数：8671次

Hadoop的distcp命令可以实现将文件从1个hdfs文件系统中拷贝到另外1个文件系统中，以下所示：

$ bin/hadoop distcp -overwrite hdfs://123.123.23.111:9000/hsd/t_url hdfs://123.123.23.156:9000/data/t_url

正常情况下应当出现以下运行结果：

Java HotSpot(TM) 64-Bit Server VM warning: Insufficient space for shared memory file: /tmp/hsperfdata_hugetable/16744 Try using the -Djava.io.tmpdir= option to select an alternate temp location. 15/04/29 20:35:07 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[hdfs://192.168.34.135:9000/zyx/t_url], targetPath=hdfs://192.168.34.156:9000/data/t_url, targetPathExists=false, preserveRawXattrs=false} 15/04/29 20:35:07 INFO client.RMProxy: Connecting to ResourceManager at compute⑵3-06.local/192.168.34.135:8032 15/04/29 20:35:08 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb 15/04/29 20:35:08 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor 15/04/29 20:35:08 WARN conf.Configuration: bad conf file: element not <property> 15/04/29 20:35:08 WARN conf.Configuration: bad conf file: element not <property> 15/04/29 20:35:08 INFO client.RMProxy: Connecting to ResourceManager at compute⑵3-06.local/192.168.34.135:8032 15/04/29 20:35:09 INFO mapreduce.JobSubmitter: number of splits:21 15/04/29 20:35:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1429262156603_0032 15/04/29 20:35:10 INFO impl.YarnClientImpl: Submitted application application_1429262156603_0032 15/04/29 20:35:10 INFO mapreduce.Job: The url to track the job: http://compute⑵3-06.local:8088/proxy/application_1429262156603_0032/ 15/04/29 20:35:10 INFO tools.DistCp: DistCp job-id: job_1429262156603_0032 15/04/29 20:35:10 INFO mapreduce.Job: Running job: job_1429262156603_0032 15/04/29 20:35:21 INFO mapreduce.Job: Job job_1429262156603_0032 running in uber mode : false 15/04/29 20:35:21 INFO mapreduce.Job: map 0% reduce 0% 15/04/29 20:35:32 INFO mapreduce.Job: map 10% reduce 0% 15/04/29 20:35:33 INFO mapreduce.Job: map 18% reduce 0% 15/04/29 20:35:34 INFO mapreduce.Job: map 25% reduce 0% ……

但是我在运行的进程中出现了Permission denied毛病，具体以下：

Error: java.io.IOException: File copy failed: hdfs://192.168.34.135:9000/zyx/t_url/000031_0 --> hdfs://192.168.34.156:9000/data/000031_0 at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:284) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:252) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://192.168.34.135:9000/zyx/t_url/000031_0 to hdfs://192.168.34.156:9000/data/000031_0 at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:280) ... 10 more Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=hugetable, access=WRITE, inode="/data":root:supergroup:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:257) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:238) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:179) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5584) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5566) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5540) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2282) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2235) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2188) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:505) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:354) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)

从命令中可以很明显的看出来，是权限不够，“supergroup:drwxr-x-r-x”，通过hadoop fs -ls /来查看，可见对组外的其他用户的确没有写权限（drwxr-xr-x的最后3个是组外的其他用户权限，目前只有r:read和x:executable两个权限）。可见用户对hdfs文件没有写权限。