如何更快地从 SQL 中获取大尺寸数据

问题描述

我正在努力加快从 sql 获取大型数据的速度。

这是我的系统信息。

APP

示例代码

OS: Linux
Language: python3
sql library: pymssql

样本数据有 3 万行。

msg= "SELECT TAG_CODE,UI_X,UI_Y,INSERT_TIME FROM dbo.R_TAG_HIST WHERE "
        msg+= "CREATE_TIME > '" + self.start_time + \
            "' AND CREATE_TIME < '" + self.end_time + "' "
        msg += "AND TAG_CODE IN " + \
            str(self.tag_ids).replace("[","(").replace("]",")")
        msg+= " ORDER BY TAG_CODE"

def receive_all_tables(self,msg):
    try:
        # check connection
        try:
            hasattr(self.conn,'_conn')
        except:
            self.connect_db()

        with self.conn.cursor() as cursor:
            cursor.execute(msg)             
            tables = cursor.fetchall()      
            self.conn.commit()                  
        return tables

    except Exception as e:
        exc_type,exc_obj,exc_tb = sys.exc_info()
        print("fail to receive query.",exc_type,exc_tb.tb_lineno,e)

def result_iterator(self,cursor,arraysize=1000):
        # 'iterator using fetchmany and consumes less memory'
        while True:
            results = cursor.fetchmany(arraysize)
            if not results:
                break
            for result in results:
                yield result

def receive_all_tables_by_iterator(self,'_conn')
        except:
            self.connect_db()

        tables=[]
        with self.conn.cursor() as cursor:
            cursor.execute(msg)             
            for result in self.result_iterator(cursor) :
                tables.append(result)       
            # self.conn.commit()                   
        return tables

我想减少获取数据的时间。
我想知道另一种从数据库接收大数据的好方法。

请帮帮我:)

解决方法

您可以尝试 multiprocessing 或 spark 以更快地查询大数据

1. Fastest way to read huge MySQL table in python
2. https://spark.apache.org/docs/1.5.2/sql-programming-guide.html

,

为了加快查询速度，您可以做的一件事是在 CREATE_TIME 和 TAG_CODE 列上添加索引（如果它们没有）。