减少create上的数据库查询量

问题描述

| 目前，我正在开发一个必须解析大型XML文件并使用其中的数据在数据库中创建实体的应用程序。创建实体的代码是控制器调用的以下部分：

def create
    parser = EuroPassVacancy.new params[:vacancy][:file].tempfile.path
    ids = parser.collect_vacancy_ids
    ids.each do |vacancy_id|
      country = Country.find_or_create_by_code(:code => parser.country_code(vacancy_id),:name => parser.country_code(vacancy_id))
      company = Company.find_or_create_by_name(:name => parser.employer_name(vacancy_id),:address => parser.employer_address(vacancy_id),:contact => parser.employer_contact(vacancy_id),:phone => parser.employer_phone(vacancy_id),:email => parser.employer_email(vacancy_id),:fax => parser.employer_fax(vacancy_id),:country => country)
      Vacancy.create(:pes_id => parser.pes_id(vacancy_id),:title => parser.title(vacancy_id),:description => parser.description(vacancy_id),:country => country,:region_code => parser.region_code(vacancy_id),:company_id => company,:isco_code => parser.isco_code(vacancy_id),:nace_code => parser.nace_code(vacancy_id))
    end
    redirect_to :action => :index
  end

这是一个非常沉重的循环，但是我有足够的信心相信解析会顺利进行。当前困扰我的是为每个create语句生成的查询数量。以下是空缺的2个insert语句的一个小副本：

AREL (0.0ms)  INSERT INTO `vacancies` (`country_id`,`pes_id`,`title`,`region_code`,`isco_code`,`created_at`,`updated_at`,`description`,`company_id`,`nace_code`) VALUES (9,50,\'Konstruktér strojní,Strojírenští technici projektanti,konstruktéři\',\'RL041\',3118,\'2011-05-25 13:19:16\',\'ÚSO - strojní. Konstrukce nástrojů na tváření a lisování kovů,forem na lisování plastů. Znalost tech. a právních norem,ISO 9001,14001 souvisejících s výrobním programem v rozsahu potřebném pro vývoj produktu a konstrukci výrobních prostředků - lisování a tváření kovů,plastů,montážních přípravků,PC - konstruk�ní SW - CAD,3-D (Solid Edge výhodou),OFFICE Outlook,NJ/AJ uživatelská úroveň,ŘP sk. B.\',1,0)
  SQL (31.2ms)  COMMIT
  Country Load (0.0ms)  SELECT `countries`.* FROM `countries` WHERE `countries`.`code` = \'cy\' LIMIT 1
  SQL (0.0ms)  BEGIN
  AREL (0.0ms)  INSERT INTO `countries` (`updated_at`,`name`,`code`,`created_at`) VALUES (\'2011-05-25 13:19:18\',\'cy\',\'2011-05-25 13:19:18\')
  SQL (63.0ms)  COMMIT
  Company Load (0.0ms)  SELECT `companies`.* FROM `companies` WHERE `companies`.`name` = \'ΚΛΕΙΤΟΣ ΓΕΩΡΓΙΟΥ & ΥΙΟΣ ΛΙΜΙΤΕΔ\' LIMIT 1
  SQL (0.0ms)  BEGIN
  SQL (0.0ms)  SELECT 1 FROM `companies` WHERE `companies`.`name` = \'ΚΛΕΙΤΟΣ ΓΕΩΡΓΙΟΥ & ΥΙΟΣ ΛΙΜΙΤΕΔ\' AND `companies`.`country_id` = 10 AND (`companies`.`name` = BINARY \'ΚΛΕΙΤΟΣ ΓΕΩΡΓΙΟΥ & ΥΙΟΣ ΛΙΜΙΤΕΔ\') LIMIT 1
  SQL (0.0ms)  ROLLBACK
  SQL (0.0ms)  BEGIN
  SQL (0.0ms)  ROLLBACK
  Country Load (0.0ms)  SELECT `countries`.* FROM `countries` WHERE `countries`.`code` = \'cy\' LIMIT 1
  Company Load (0.0ms)  SELECT `companies`.* FROM `companies` WHERE `companies`.`name` = \'ΠΑ�ΙΚΚΟΣ ΠΙΣΣΟΥΡΙΟΣ ΚΑΙ ΑΔΕΛΦΟΣ ΚΟ ΛΤΔ\' LIMIT 1
  SQL (0.0ms)  BEGIN
  SQL (15.6ms)  SELECT 1 FROM `companies` WHERE `companies`.`name` = \'ΠΑ�ΙΚΚΟΣ ΠΙΣΣΟΥΡΙΟΣ ΚΑΙ ΑΔΕΛΦΟΣ ΚΟ ΛΤΔ\' AND `companies`.`country_id` = 10 AND (`companies`.`name` = BINARY \'ΠΑ�ΙΚΚΟΣ ΠΙΣΣΟΥΡΙΟΣ ΚΑΙ ΑΔΕΛΦΟΣ ΚΟ ΛΤΔ\') LIMIT 1
  SQL (0.0ms)  ROLLBACK
  SQL (0.0ms)  BEGIN
  SQL (0.0ms)  ROLLBACK
  CACHE (0.0ms)  SELECT `countries`.* FROM `countries` WHERE `countries`.`code` = \'cy\' LIMIT 1
  Company Load (0.0ms)  SELECT `companies`.* FROM `companies` WHERE `companies`.`name` = \'THE DELHI PALACE INDIAN RESTAURANT LTD\' LIMIT 1
  SQL (0.0ms)  BEGIN
  SQL (0.0ms)  SELECT 1 FROM `companies` WHERE `companies`.`name` = \'THE DELHI PALACE INDIAN RESTAURANT LTD\' AND `companies`.`country_id` = 10 AND (`companies`.`name` = BINARY \'THE DELHI PALACE INDIAN RESTAURANT LTD\') LIMIT 1
  SQL (0.0ms)  ROLLBACK
  SQL (0.0ms)  BEGIN
  SQL (0.0ms)  ROLLBACK
  CACHE (0.0ms)  SELECT `countries`.* FROM `countries` WHERE `countries`.`code` = \'cy\' LIMIT 1
  Company Load (0.0ms)  SELECT `companies`.* FROM `companies` WHERE `companies`.`name` = \'EURES ADVISOR,STELLA PAPADOPOULOU APPIOU\' LIMIT 1
  SQL (0.0ms)  BEGIN
  SQL (0.0ms)  SELECT 1 FROM `companies` WHERE `companies`.`name` = \'EURES ADVISOR,STELLA PAPADOPOULOU APPIOU\' AND `companies`.`country_id` = 10 AND (`companies`.`name` = BINARY \'EURES ADVISOR,STELLA PAPADOPOULOU APPIOU\') LIMIT 1
  AREL (0.0ms)  INSERT INTO `companies` (`email`,`phone`,`country_id`,`address`,`contact`,`fax`,`name`) VALUES (\'[e-mail]\',\'00357 22 403000\',10,\'2011-05-25 13:20:13\',\'MOUSEIOU STREET 3 1102,ΛΕΥΚΩΣΙΑ\',\'STELLA PAPADOPOULOU APPIOU\',\'00357 22 873170/1\',\'EURES ADVISOR,STELLA PAPADOPOULOU APPIOU\')
  SQL (31.2ms)  COMMIT
  SQL (0.0ms)  BEGIN
  SQL (0.0ms)  ROLLBACK
  Country Load (0.0ms)  SELECT `countries`.* FROM `countries` WHERE `countries`.`code` = \'cy\' LIMIT 1
  Company Load (0.0ms)  SELECT `companies`.* FROM `companies` WHERE `companies`.`name` = \'EURES ASSISTANT,NORA LANDA\' LIMIT 1
  SQL (0.0ms)  BEGIN
  SQL (0.0ms)  SELECT 1 FROM `companies` WHERE `companies`.`name` = \'EURES ASSISTANT,NORA LANDA\' AND `companies`.`country_id` = 10 AND (`companies`.`name` = BINARY \'EURES ASSISTANT,NORA LANDA\') LIMIT 1
  AREL (0.0ms)  INSERT INTO `companies` (`email`,`name`) VALUES (\'[e-mail]\',\'00357 25 827327\',\'2011-05-25 13:20:29\',\'67 FRANKLIN ROOSEVELT AV 3011,ΛΕΜΕΣΟΣ\',\'NOT AVAILABLE\',\'00357 25 306563\',\'EURES ASSISTANT,NORA LANDA\')
  SQL (31.2ms)  COMMIT
  SQL (0.0ms)  BEGIN
  SQL (0.0ms)  ROLLBACK
  Country Load (0.0ms)  SELECT `countries`.* FROM `countries` WHERE `countries`.`code` = \'cy\' LIMIT 1
  Company Load (0.0ms)  SELECT `companies`.* FROM `companies` WHERE `companies`.`name` = \'ΕΛΕ�Α ΤΖΙΟΒΑ�Η\' LIMIT 1
  SQL (0.0ms)  BEGIN
  SQL (0.0ms)  SELECT 1 FROM `companies` WHERE `companies`.`name` = \'ΕΛΕ�Α ΤΖΙΟΒΑ�Η\' AND `companies`.`country_id` = 10 AND (`companies`.`name` = BINARY \'ΕΛΕ�Α ΤΖΙΟΒΑ�Η\') LIMIT 1
  AREL (0.0ms)  INSERT INTO `companies` (`email`,`name`) VALUES (\'[e-mail]\',\'0035723833780\',\'2011-05-25 13:20:45\',\'ΔΑΦ�ΗΣ 11-13 5314 AMMOHOSTOS\',\'MRS GEORGIA MASIA\',\'0035723833177\',\'ΕΛΕ�Α ΤΖΙΟΒΑ�Η\')
  SQL (31.2ms)  COMMIT
  SQL (0.0ms)  BEGIN
  SQL (0.0ms)  ROLLBACK
  Country Load (0.0ms)  SELECT `countries`.* FROM `countries` WHERE `countries`.`code` = \'cy\' LIMIT 1
  Company Load (0.0ms)  SELECT `companies`.* FROM `companies` WHERE `companies`.`name` = \'EURES ADVISOR,MINAS HADJICONSTANTI\' LIMIT 1
  SQL (0.0ms)  BEGIN
  SQL (0.0ms)  SELECT 1 FROM `companies` WHERE `companies`.`name` = \'EURES ADVISOR,MINAS HADJICONSTANTI\' AND `companies`.`country_id` = 10 AND (`companies`.`name` = BINARY \'EURES ADVISOR,MINAS HADJICONSTANTI\') LIMIT 1
  AREL (0.0ms)  INSERT INTO `companies` (`email`,`name`) VALUES (\'[e-mail]\',\'00357 24 805328\',\'2011-05-25 13:21:01\',\'P.O.BOX 40136 6301,ΛΑΡ�ΑΚΑ\',\'MINAS HADJICONSTANTI\',\'00357 24 304532\',MINAS HADJICONSTANTI\')
  SQL (62.4ms)  COMMIT
  SQL (0.0ms)  BEGIN
  SQL (1.0ms)  ROLLBACK
  Country Load (0.0ms)  SELECT `countries`.* FROM `countries` WHERE `countries`.`code` = \'it\' LIMIT 1
  Company Load (0.0ms)  SELECT `companies`.* FROM `companies` WHERE `companies`.`name` = \'SAILER ANDREAS\' LIMIT 1
  SQL (0.0ms)  BEGIN
  SQL (0.0ms)  SELECT 1 FROM `companies` WHERE `companies`.`name` = \'SAILER ANDREAS\' AND `companies`.`country_id` = 8 AND (`companies`.`name` = BINARY \'SAILER ANDREAS\') LIMIT 1
  AREL (0.0ms)  INSERT INTO `companies` (`email`,`name`) VALUES (\'[e-mail]\',\'[phone]\',8,\'2011-05-25 13:21:18\',\'A.Duerer-Str. 20 - Via Duerer 20 39100-Bozen - Bolzano\',\'Giulia Morello\',\'[phone]\',\'SAILER ANDREAS\')
  SQL (30.0ms)  COMMIT
  SQL (0.0ms)  BEGIN
  AREL (0.0ms)  INSERT INTO `vacancies` (`country_id`,`nace_code`) VALUES (8,64,\'ricerca di ingegnere e architetto per studio di ingegneria\',\'R3311\',4190,\'2011-05-25 13:21:24\',\'Studio di ingegneria cerca,anche per subito,un ingegnere e un architetto,per la progettazione architettonica e strutturale,stima e contabilità di opere,il tutto nel settore delle costruzioni civili e delle opere speciali . E\\\' richiesta ottima conoscenza di autocad,pacchetto office e programmi di strutture e contabilità. Sede di lavoro: Bolzano centro: si chiede gentilmente che venga inviato un curriculum professionale con foto allegata.\',74)
  SQL (62.4ms)  COMMIT

对于进行中的国家和公司，似乎有很多选择查询要比我想要的要多。我想知道是否可以对代码进行优化以减少这些查询。     

解决方法

破解此漏洞的最简单方法是缓存您执行的查找或创建。例如，您要一遍又一遍地选择同一国家/地区，这已经很浪费了。 您可以通过定义一个辅助方法来做到这一点：

class Country < ActiveRecord::Base
  def self.resolve_country(options)
    @resolved_countries ||= { }

    @resolved_countries[options[:code]] ||= Country.find_or_create_by_code(options)
  end
end

这样只会将您的国家/地区载入一次，如果将此模式应用于其他模型，则可以大大加快您的载入速度。 要记住的一件事是何时可能需要出于测试目的而重置这些缓存。添加方法来执行此操作总是一个好主意，并不难：

def self.reset_cache!
  @resolved_countries = nil
end

可以将其放入模型的单元测试的

setup

方法中。