问题描述
import org.apache.spark._
import org.apache.spark.SparkContext._
import com.datastax.spark.connector._
import com.datastax.spark.connector.cql.CassandraConnector
val conf = new SparkConf()
.setMaster("local[*]")
.setAppName("XXXX")
.set("spark.cassandra.connection.host","cassandra.us-east-2.amazonaws.com")
.set("spark.cassandra.connection.port","9142")
.set("spark.cassandra.auth.username","XXXXX")
.set("spark.cassandra.auth.password","XXXXX")
.set("spark.cassandra.connection.ssl.enabled","true")
.set("spark.cassandra.connection.ssl.trustStore.path","/home/nihad/.cassandra/cassandra_truststore.jks")
.set("spark.cassandra.connection.ssl.trustStore.password","XXXXX")
.set("spark.cassandra.output.consistency.level","LOCAL_QUORUM")
val connector = CassandraConnector(conf)
val session = connector.openSession()
sesssion.execute("""INSERT INTO "covid19".delta_by_states (state_code,state_value,date ) VALUES ('kl',5,'2020-03-03');""")
session.close()
我试图使用我本地系统中设置的Spark App将数据写入AWS Cassandra Keyspace。
问题是当我执行上述代码时,出现如下异常:
“ com.datastax.oss.driver.api.core.servererrors.InvalidQueryException: 此操作不支持一致性级别LOCAL_ONE。 支持的一致性级别为:LOCAL_QUORUM“
从上面的代码中可以看到,我已经在spark conf中将cassandra.output.consistency.level设置为LOCAL_QUORUM。我也在使用datastax cassandra驱动程序。
但是当我从AWS Cassandra读取数据时,它可以正常工作。我也在AWS Keyspace cqlsh中尝试了相同的INSERT命令。那里也很好。因此查询有效。
有人可以帮助我如何通过datastax.CassandraConnector设置一致性吗?
解决方法
破解。
不是通过spark配置来设置cassandra一致性。我在src / main / resources目录中创建了一个application.conf文件。
datastax-java-driver {
basic.contact-points = [ "cassandra.us-east-2.amazonaws.com:9142"]
advanced.auth-provider{
class = PlainTextAuthProvider
username = "serviceUserName"
password = "servicePassword"
}
basic.load-balancing-policy {
local-datacenter = "us-east-2"
}
advanced.ssl-engine-factory {
class = DefaultSslEngineFactory
truststore-path = "yourPath/.cassandra/cassandra_truststore.jks"
truststore-password = "trustorePassword"
}
basic.request.consistency = LOCAL_QUORUM
basic.request.timeout = 5 seconds
}
并创建了如下所示的cassandra会话
import com.datastax.oss.driver.api.core.config.DriverConfigLoader
import com.datastax.oss.driver.api.core.CqlSession
val loader = DriverConfigLoader.fromClassPath("application.conf")
val session = CqlSession.builder().withConfigLoader(loader).build()
sesssion.execute("""INSERT INTO "covid19".delta_by_states (state_code,state_value,date ) VALUES ('kl',5,'2020-03-03');""")
终于成功了。无需弄乱spark配置 驱动程序配置https://docs.datastax.com/en/drivers/java/4.0/com/datastax/oss/driver/api/core/config/DriverConfigLoader.html#fromClasspath-java.lang.String-
的文档datastax配置文档https://docs.datastax.com/en/developer/java-driver/4.6/manual/core/configuration/reference/