使用Apache Ozone FileSystem API会导致错误

问题描述

我想将文件上传到Apache Ozone。我使用FileSystem界面。 Java代码如下

java代码：

        Configuration conf = new Configuration();
        conf.addResource(new OzoneConfiguration());

        System.setProperty("HADOOP_USER_NAME","work");
        FileSystem fs = FileSystem.get(conf);   //"fs.defaultFS" o3fs://bucket1.volume1/

        String objectName = UUID.randomUUID().toString();
        byte[] objectData = readFile(objectName);
        Path newfilePath = new Path("/" + objectName);

        FSDataOutputStream outputStream = fs.create(newfilePath,(short) 3);
        outputStream.write(objectData);
        outputStream.flush();

        IoUtils.closeStream(outputStream);
        fs.close();

我将配置文件放在Resources文件夹中。

core-site.xml的配置为：

<property>
        <name>fs.o3fs.impl</name>
        <value>org.apache.hadoop.fs.ozone.BasicOzoneFileSystem</value>
    </property>
    <property>
        <name>fs.AbstractFileSystem.o3fs.impl</name>
        <value>org.apache.hadoop.fs.ozone.OzFs</value>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>o3fs://bucket1.volume1/</value>
    </property>

例外是：

Exception in thread "main" java.lang.NullPointerException: client is null
    at java.util.Objects.requireNonNull(Objects.java:228)
    at org.apache.hadoop.hdds.scm.XceiverClientRatis.getClient(XceiverClientRatis.java:201)
    at org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequestAsync(XceiverClientRatis.java:227)
    at org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommandAsync(XceiverClientRatis.java:305)
    at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunkAsync(ContainerProtocolCalls.java:315)
    at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunkToContainer(BlockOutputStream.java:599)
    at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunk(BlockOutputStream.java:452)
    at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.handleFlush(BlockOutputStream.java:463)
    at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.flush(BlockOutputStream.java:429)
    at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.flush(BlockOutputStreamEntry.java:137)
    at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleStreamAction(KeyOutputStream.java:489)
    at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleFlushOrClose(KeyOutputStream.java:455)
    at org.apache.hadoop.ozone.client.io.KeyOutputStream.flush(KeyOutputStream.java:428)
    at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.flush(OzoneFSOutputStream.java:51)
    at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
    at java.io.DataOutputStream.flush(DataOutputStream.java:123)
    at FSTest.main(FSTest.java:28)

pom.xml

<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-ozone-client -->
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-ozone-client</artifactId>
    <version>0.5.0-beta</version>
</dependency>


<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-ozone-filesystem-lib-legacy -->
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-ozone-filesystem-lib-legacy</artifactId>
    <version>0.5.0-beta</version>
</dependency>

解决方法

https://docs.alluxio.io/os/user/edge/en/ufs/Ozone.html

参考本文档，臭氧-0.5.0-beta有一个与ScmClientConfig相关的错误，因此您必须将以下配置添加到臭氧-site.xml中

<configuration>
  <property>
    <name>ozone.scm.names</name>
    <value>localhost</value>
  </property>
  <property>
    <name>scm.container.client.max.size</name>
    <value>256</value>
  </property>
  <property>
    <name>scm.container.client.idle.threshold</name>
    <value>10s</value>
  </property>
  <property>
    <name>hdds.ratis.raft.client.rpc.request.timeout</name>
    <value>60s</value>
  </property>
  <property>
    <name>hdds.ratis.raft.client.async.outstanding-requests.max</name>
    <value>32</value>
  </property>
  <property>
    <name>hdds.ratis.raft.client.rpc.watch.request.timeout</name>
    <value>180s</value>
  </property>
</configuration>

希望这可以为您提供帮助。

java java ozone