在Scala Spark中编写简单的Twitter流式程序时出错-短时间内登录尝试过多

问题描述

这是我的程序：

import org.apache.spark.streaming.{Seconds,StreamingContext}
import twitter4j.TwitterFactory
import twitter4j.conf.ConfigurationBuilder

object TwitterHashtagStreaming {
  def main(args : Array[String]) {

    val sparkConf = new SparkConf().setAppName("TwitterHashtagStreaming").setMaster("local[*]")
    val ssc = new StreamingContext(sparkConf,Seconds(10))
    println(ssc.getState())

    if (!Logger.getRootLogger.getAllAppenders.hasMoreElements) {
      Logger.getRootLogger.setLevel(Level.WARN)
    }
    // (1) config work to create a twitter object
    val cb = new ConfigurationBuilder()
    cb.setDebugEnabled(true)
      .setOAuthConsumerKey("xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
      .setOAuthConsumerSecret("xxxxxxxxxxxxxxxxxxxxxxxxxx")
      .setOAuthAccessToken("xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
      .setOAuthAccessTokenSecret("xxxxxxxxxxxxxxxxxxxxxxxx")
    val tf = new TwitterFactory(cb.build())
    val twitter = tf.getInstance().getAuthorization()
    println(twitter)

    val stream = TwitterUtils.createStream(ssc,Some(twitter))
    stream.print()

    ssc.start()
    ssc.awaitTerminationOrTimeout(10000)
    ssc.stop()
  }
}

这是build.sbt：

name := "Test1"
    
version := "0.1"

scalaVersion := "2.11.12"
scalacOptions += "-target:jvm-1.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.0"

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.3.0"
libraryDependencies += "org.apache.bahir" %% "spark-streaming-twitter" % "2.0.0"


libraryDependencies += "com.typesafe" % "config" % "1.3.0"
libraryDependencies += "org.twitter4j" % "twitter4j-core" % "3.0.3"
libraryDependencies += "org.twitter4j" % "twitter4j-stream" % "3.0.3"

运行程序时出现错误： 有关讨论可以在Internet上找到： http://www.google.co.jp/search?q=4d13a6ae或 http://www.google.co.jp/search?q=b77019f1

TwitterException{exceptionCode=[4d13a6ae-b77019f1],statusCode=420,message=null,code=-1,retryAfter=-1,rateLimitStatus=null,version=4.0.4}
20/10/28 14:30:04 INFO TwitterStreamImpl: Waiting for 10000 milliseconds
20/10/28 14:30:04 INFO TwitterStreamImpl: 420:Returned by the Search and Trends API when you are being rate limited (https://dev.twitter.com/docs/rate-limiting).
Returned by the Streaming API:
 Too many login attempts in a short period of time.
 Running too many copies of the same application authenticating with the same account name.
Easy there,Turbo. Too many requests recently. Enhance your calm.


java.lang.AbstractMethodError
    at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99)
    at org.apache.spark.streaming.twitter.TwitterReceiver.initializeLogIfNecessary(TwitterInputDStream.scala:60)
    at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
    at org.apache.spark.streaming.twitter.TwitterReceiver.log(TwitterInputDStream.scala:60)
    at org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
    at org.apache.spark.streaming.twitter.TwitterReceiver.logInfo(TwitterInputDStream.scala:60)
    at org.apache.spark.streaming.twitter.TwitterReceiver.onStop(TwitterInputDStream.scala:106)
    at org.apache.spark.streaming.receiver.ReceiverSupervisor.stopReceiver(ReceiverSupervisor.scala:170)
    at org.apache.spark.streaming.receiver.ReceiverSupervisor$$anonfun$restartReceiver$1.apply$mcV$sp(ReceiverSupervisor.scala:194)
    at org.apache.spark.streaming.receiver.ReceiverSupervisor$$anonfun$restartReceiver$1.apply(ReceiverSupervisor.scala:189)
    at org.apache.spark.streaming.receiver.ReceiverSupervisor$$anonfun$restartReceiver$1.apply(ReceiverSupervisor.scala:189)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
20/10/28 14:30:04 INFO TwitterStreamImpl: 420:Returned by the Search and Trends API when you are being rate limited (https://dev.twitter.com/docs/rate-limiting).
Returned by the Streaming API:
 Too many login attempts in a short period of time.
 Running too many copies of the same application authenticating with the same account name.
Easy there,Turbo. Too many requests recently. Enhance your calm.

您能帮我解决此错误吗？ 将此语句添加到程序中时，我收到错误消息- val stream = TwitterUtils.createStream(ssc,Some(twitter))

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）