问题描述
我已按照此处所述在 AWS EC2 实例上安装了 Apache Superset - https://superset.apache.org/docs/installation/installing-superset-from-scratch。
当我运行 superset load_examples 命令时，出现以下错误。这在我的本地计算机上运行良好，但由于某种原因在 EC2 上不起作用。错误提示“变量太多”，但只有 7 个变量。
sql Alchemy 版本如下：
sqlAlchemy==1.3.22
sqlAlchemy-Utils==0.36.8
错误日志：
Traceback (most recent call last):
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/sqlalchemy/engine/base.py",line 1277,in _execute_context
cursor,statement,parameters,context
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/sqlalchemy/engine/default.py",line 609,in do_execute
cursor.execute(statement,parameters)
sqlite3.OperationalError: too many sql variables
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ec2-user/superset/venv/bin/superset",line 8,in <module>
sys.exit(superset())
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/click/core.py",line 829,in __call__
return self.main(*args,**kwargs)
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/flask/cli.py",line 586,in main
return super(FlaskGroup,self).main(*args,**kwargs)
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/click/core.py",line 782,in main
rv = self.invoke(ctx)
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/click/core.py",line 1259,in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/click/core.py",line 1066,in invoke
return ctx.invoke(self.callback,**ctx.params)
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/click/core.py",line 610,in invoke
return callback(*args,**kwargs)
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/click/decorators.py",line 21,in new_func
return f(get_current_context(),*args,line 426,in decorator
return __ctx.invoke(f,**kwargs)
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/superset/cli.py",line 172,in load_examples
load_examples_run(load_test_data,only_Metadata,force)
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/superset/cli.py",line 116,in load_examples_run
examples.load_birth_names(only_Metadata,force)
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/superset/examples/birth_names.py",line 94,in load_birth_names
load_data(tbl_name,database,sample=sample)
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/superset/examples/birth_names.py",line 78,in load_data
index=False,File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/pandas/core/generic.py",line 2615,in to_sql
method=method,File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/pandas/io/sql.py",line 598,line 1406,in to_sql
raise err
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/pandas/io/sql.py",line 1398,in to_sql
table.insert(chunksize,method=method)
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/pandas/io/sql.py",line 830,in insert
exec_insert(conn,keys,chunk_iter)
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/pandas/io/sql.py",line 758,in _execute_insert_multi
conn.execute(self.table.insert(data))
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/sqlalchemy/engine/base.py",line 1011,in execute
return meth(self,multiparams,params)
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/sqlalchemy/sql/elements.py",line 298,in _execute_on_connection
return connection._execute_clauseelement(self,params)
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/sqlalchemy/engine/base.py",line 1130,in _execute_clauseelement
distilled_params,File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/sqlalchemy/engine/base.py",line 1317,in _execute_context
e,cursor,context
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/sqlalchemy/engine/base.py",line 1511,in _handle_dbapi_exception
sqlalchemy_exception,with_traceback=exc_info[2],from_=e
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/sqlalchemy/util/compat.py",line 182,in raise_
raise exception
File "/home/ec2-user/superset/venv/lib64/python3.7/site-packages/sqlalchemy/engine/base.py",parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) too many sql variables
[sql: INSERT INTO birth_names (ds,gender,name,num,state,sum_boys,sum_girls) VALUES (?,?,?),(?,?)]
[parameters: ('1965-01-01 00:00:00.000000','boy','Aaron',369,'CA','1965-01-01 00:00:00.000000','girl','Amy',494,'Andrea',607,'Andrew',933,'Angela',1066,'Anna',564,'Anthony',1928,'Barbara',922,'Bradley',395,'Brenda',961,'Brian',2233,'Bryan',542,'Carlos',401,'Carol',816,'Carrie',367,'Catherine',706,'Charles',1609,'Cheryl',1052,'Christina',848,'Christine',1450,'Christopher',2322,'Cindy',842,'Craig',874,'Cynthia',1947,'Dana',546,'Daniel',2641,'David',6820,'Dawn',962,'Deborah',1610,'Debra',1153,'Denise',1423,'Dennis',801,'Diana',864,'Diane',947,'Donald',1388,'Donna',1270,'Douglas',972,'Edward',1280,'Elizabeth',1676,'Eric',2241,'Frank',930,'Gary',1441,'George',924,'Gregory',1669,'Heather',404,'Jack',361,'Jacqueline',690,'James',5060,.....)]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
解决方法
奇怪的是 sqlite3 的最大参数值是 999，并且只能通过重新编译来更改它 (SQLITE_MAX_VARIABLE_NUMBER)。
请注意，Apache superset 不主动支持 sqlite，不建议将此引擎用于生产。看看：https://www.sqlite.org/whentouse.html
If many threads and/or processes need to write the database at the same instant (and they cannot queue up and take turns) then it is best to select a database engine that supports that capability,which always means a client/server database engine.