版权声明：本文为博主原创文章，未经博主同意不得转载。安金龙 的博客。 https://blog.csdn.net/smile0198/article/details/37573081

1、从HDFS中读取数据

Configuration conf = getConf();  Path path = new Path(pathstr);   FileSystem fs = FileSystem.get(conf);   FSDataInputStream fsin= fs.open(path );    BufferedReader br =null;   String line ;   try{    br = new BufferedReader(new InputStreamReader(fsin));       while ((line = br.readLine()) != null) {         System.out.println(line);        }    }finally{    br.close();   }

2、写HDFS

Configuration conf = getConf();  Path path = new Path(mid_sort);   FileSystem fs = FileSystem.get(conf);   FSDataOutputStream out = fs.create(resultpath);  out.write(sb.toString().getBytes());  out.close();

3、遍历文件夹 获取文件 全路径

/**  * 得到一个文件夹(不包含子文件夹)下的全部名字匹配上pattern的文件名称  * @param fs  * @param folderPath  * @param pattern 用于匹配文件名称的正则  * @return  * @throws IOException  */ public static List
      
        getFilesUnderFolder(FileSystem fs, Path folderPath, String pattern) throws IOException {  List
       
         paths = new ArrayList
        
         ();  if (fs.exists(folderPath)) {   FileStatus[] fileStatus = fs.listStatus(folderPath);   for (int i = 0; i < fileStatus.length; i++) {    FileStatus fileStatu = fileStatus[i];    if (!fileStatu.isDir()) {//仅仅要文件     Path oneFilePath = fileStatu.getPath();     if (pattern == null) {      paths.add(oneFilePath);     } else {      if (oneFilePath.getName().contains(pattern)) {       paths.add(oneFilePath);      }     }      }   }  }  return paths; }
        
       
      

4、追加数据 append

public static boolean appendRTData(String hdfsFile, String appendFile) {    boolean flag = false;    Configuration conf = new Configuration();    FileSystem fs = null;    try {      fs = FileSystem.get(URI.create(hdfsFile), conf);      InputStream in = new BufferedInputStream(new FileInputStream(appendFile));      OutputStream out = fs.append(new Path(hdfsFile));      IOUtils.copyBytes(in, out, 4096, true);    } catch (IOException e) {      e.printStackTrace();    }    return flag;  }

***********************************************************************************************************************************************

***********************************************************************************************************************************************

异常信息

1、Exception in thread "main" java.lang.IllegalArgumentException: java.net.UnknownHostException: ns6

原因是没有载入hdfs的配置信息，须要加入以下的代码：

conf.addResource(new Path("/xxxx/hdfs-site.xml"));//path是配置文件地址

假设配置了环境变量能够在不同的机器上使用:

conf.addResource(new Path(System.getenv("HADOOP_CONF") + "/hdfs-site.xml"));