在使用pandas和dask读取sqlite表时,当我从带有存储为NUMERIC数据类型的日期时间(ISO格式的字符串)的sqlite表中进行选择时,遇到了sqlAlchemy的一些意外行为. sqlAlchemy原始SQL查询工作正常,但是使用从反射构造的可选对象的查询失败.这两个查询似乎是等效的.
我在下面粘贴了一个示例以及回溯.有人可以解释示例中第三个查询的问题吗？
用NUMERIC日期时间设置表格：
import sqlalchemy as sa
from sqlalchemy import text
connString = "sqlite:///c:\\temp\\test.db"
engine = sa.create_engine(connString)
conn = engine.connect()
conn.execute("create table testtable (uid INTEGER Primary Key, datetime NUMERIC)")
conn.execute("insert into testtable values (1, '2017-08-03 01:11:31')")
print(conn.execute('PRAGMA table_info(testtable)').fetchall())
# [(0, 'uid', 'INTEGER', 0, None, 1), (1, 'datetime', 'NUMERIC', 0, None, 0)]
使用原始SQL查询的工作原理：
resultList1 = conn.execute("SELECT testtable.uid, testtable.datetime \nFROM testtable").fetchall()
print(resultList1)
# [(1, '2017-08-03 01:11:31')]
使用此可选作品进行查询：
resultList2 = conn.execute(sa.sql.select(columns=[text('uid'),text('datetime')]).select_from(text('testtable'))).fetchall()
print(resultList2)
# [(1, '2017-08-03 01:11:31')]
使用此可选查询失败：
m = sa.MetaData()
table = sa.Table('testtable', m, autoload=True, autoload_with=engine)
selectble = sa.sql.select(table.columns).select_from(table)
print(selectble.compile().string)
# note: same raw sql query as above
# "SELECT testtable.uid, testtable.datetime \nFROM testtable"
resultList3 = conn.execute(sa.sql.select(table.columns).select_from(table)).fetchall()
# SAWarning: Dialect sqlite+pysqlite does *not* support Decimal objects natively...
print(resultList3)
conn.close()
错误：
Traceback (most recent call last):
File "<ipython-input-20-188c84a35d95>", line 1, in <module>
print(resultList3)
File "c:\program files\python36\lib\site-packages\sqlalchemy\engine\result.py", line 156, in __repr__
return repr(sql_util._repr_row(self))
File "c:\program files\python36\lib\site-packages\sqlalchemy\sql\util.py", line 329, in __repr__
", ".join(trunc(value) for value in self.row),
TypeError: must be real number, not str
解决方法:
sqlite与大多数sql数据库的类型系统有很大不同：它使用dynamic typing,在conversion之后,您输入一列的类型名称确定其affinity,例如NUMERIC：
A column with NUMERIC affinity may contain values using all five storage classes. When text data is inserted into a NUMERIC column, the storage class of the text is converted to INTEGER or REAL (in order of preference) if such conversion is lossless and reversible. For conversions between TEXT and REAL storage classes, sqlite considers the conversion to be lossless and reversible if the first 15 significant decimal digits of the number are preserved. If the lossless conversion of TEXT to INTEGER or REAL is not possible then the value is stored using the TEXT storage class. No attempt is made to convert NULL or BLOB values.
由于您插入的值不可能(无损)转换为INTEGER或REAL1,因此您的值使用TEXT storage class,而sqlAlchemy / pysqlite则感到不满意,因为另一方面它期望的值是it can convert to float,即fails.
键入系统会引起其他类似的问题,例如当使用DATETIME类型名将来自CREATE TABLE … AS的结果表与来自表的SELECT反映时,该类型名将转换为NUMERIC关联.
简短的代码示例演示了该问题：
In [2]: foo = Table('foo', Metadata, Column('bar', NUMERIC))
In [3]: foo.create(engine)
CREATE TABLE foo (
bar NUMERIC
)
In [4]: engine.execute("insert into foo values ('not really a number, no')")
Out[4]: <sqlalchemy.engine.result.ResultProxy at 0x7fbcd7ee8f98>
In [5]: foo.select().execute().fetchall()
Out[5]: ---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
...
~/Work/SO/lib/python3.6/site-packages/sqlalchemy/sql/util.py in __repr__(self)
327 trunc = self.trunc
328 return "(%s%s)" % (
--> 329 ", ".join(trunc(value) for value in self.row),
330 "," if len(self.row) == 1 else ""
331 )
TypeError: must be real number, not str
1可能是sqlite pysqlite方言本机不支持Decimal的原因– neither does SQLite
