问题描述
我与黑斑羚有关
con <- DBI::dbConnect(odbc::odbc(),"impala connector",schema = "some_schema")
library(dplyr)
library(dbplyr) #I have to load both of them,if not tbl won't work
table <- tbl(con,'serverTable')
我想使用 Pearson 的 R 来及时跟踪度量的变化,作为快速而肮脏的预测模型。
在语言环境中,它工作得很好,但我在服务器上实现它时遇到了问题。 代码如下:
library(corrr)
table %>%
filter(!is.na(VAR) | VAR > -10 | VAR < -32) %>%
#VAR is the measure,and values over -10 or under -32 are already out of the threshold,I wanna intercept the subjects before that
mutate(num_date = as.numeric(as.POSIXct(date))) %>%
#to convert the date string into the number of seconds since 1970
group_by(id) %>%
#the measure is taken daily for varIoUs subjects,I am interested in isolating the subjects approaching the thresholds
mutate(corr = corrr::correlate(VAR,num_date)) %>%
ungroup() %>%
#here I calculare Pearson's R,I must specify corrr:: if not I get an error
filter(abs(corr) > 0.9) %>%
#in locale I found out that a value of 0.9 is good for isolating the subjects whose measure is approaching the thresholds
select(id) %>%
collect()
如果我运行这个,我会得到错误:
corrr::correlate(VAR,num_date) 中的错误:未找到对象“VAR”。
所以我试着用
代替那行mutate(corr = corrr::correlate(.$VAR,.$num_date)) %>%
像这样我得到错误
stats::cor(x = x,y = y,use = use,method = method) 中的错误:同时提供 'x' 和 'y' 或类似矩阵的 'x'
如果我尝试使用来自 stats 的 cor,cor(VAR,num_date)
,我会收到错误
new_result(connection@ptr,statement,immediate)中的错误:nanodbc/nanodbc.cpp:1412: HY000: [Cloudera][ImpalaODBC] (370) 查询执行期间发生查询分析错误:[HY000] : AnalysisException: some_schema .cor() 未知
像 dbplyr 不能将 cor 转换为 sql(如果我运行 show_query() 而不是 collect() ,我会看到它)
SELECT id,cor
FROM(
SELECT id,((tot_sum - (VAR_sum * date_sum / _count)) / sqrt((VAR_sq - pow(VAR_sum,2.0) / _count) * (date_sq - pow(date_sum,2.0) / _count))) AS cor
FROM (
SELECT id,sum(VAR) AS VAR_sum,sum(CAST(CAST(date AS TIMESTAMP) AS DOUBLE)) AS date_sum,sum(VAR * VAR) AS VAR_sq,sum(CAST(CAST(date AS TIMESTAMP) AS DOUBLE) * CAST(CAST(date AS TIMESTAMP) AS DOUBLE)) AS date_sq,sum(VAR * CAST(CAST(date_push AS TIMESTAMP) AS DOUBLE)) AS tot_sum,count(*) as _count
FROM (
SELECT id,VAR,date
FROM (
SELECT id,date
FROM schema
WHERE VAR IS NOT NULL) AS a
WHERE VAR < -10 OR VAR > -32) AS b
GROUP BY idur) AS c) AS d
WHERE ABS(cor) > 0.9 AND ABS(cor) <= 1
感谢这篇文章: https://chartio.com/learn/postgresql/correlation-coefficient-pearson/
解决方法
cor
不在 dplyr
可以翻译的函数列表中 - 请参阅此处:https://dbplyr.tidyverse.org/articles/sql-translation.html#known-functions
您可以在代码中尝试以下操作:
mutate(corr = translate_sql(corr(VAR,num_date)))
这应该直接转换为 CORR(VAR,num_date)
。这些翻译不适用于所有数据库类型。如果在您的情况下无法实现此功能,您可能别无选择,只能在尝试运行不可翻译的函数之前收集数据。