用于为所有现有 ActiveStorage 变体创建数据库记录的 Rake 任务

问题描述

在 Rails 6.1 中，ActiveStorage 会在第一次加载所有变体时为所有变体创建数据库记录：https://github.com/rails/rails/pull/37901

我想启用此功能，但由于我的生产 Rails 应用程序中有数以万计的文件，因此让用户在浏览站点时创建如此多的数据库记录是有问题的（并且可能很慢）。有没有办法编写一个 Rake 任务来遍历我数据库中的每个附件，并生成变体并将它们保存在数据库中？

我会在启用新的 active_storage.track_variants 配置后运行一次，然后在第一次加载时保存所有新上传的文件。

感谢您的帮助！

解决方法

这是我最终为此创建的 Rake 任务。如果你有一个较小的数据集，可以删除 Parallel 的东西，但我发现对于 70k+ 变体，在没有任何并行化的情况下执行它时速度慢得令人无法忍受。也可以忽略进度条相关代码:)

本质上，我只选取所有带有附件的模型（我手动执行此操作，如果您有大量附件，您可以以更动态的方式执行此操作），然后过滤掉不可变的模型。然后我检查每个附件并为我定义的每个尺寸生成一个变体，然后对其调用 process 以强制将其保存到数据库中。

确保捕获任务中的 MiniMagick（或 vips，如果您愿意）错误，以便错误的图像文件不会破坏一切。

# Rails 6.1 changes the way ActiveStorage works so that variants are
# tracked in the database. The intent of this task is to create the
# necessary variants for all game covers and user avatars in our database.
# This way,the user isn't creating dozens of variant records as they
# browse the site. We want to create them ahead-of-time,when we deploy
# the change to track variants.
namespace 'active_storage:vglist:variants' do
  require 'ruby-progressbar'
  require 'parallel'

  desc "Create all variants for covers and avatars in the database."
  task create: :environment do
    games = Game.joins(:cover_attachment)
    # Only attempt to create variants if the cover is able to have variants.
    games = games.filter { |game| game.cover.variable? }
    puts 'Creating game cover variants...'

    # Use the configured max number of threads,with 2 leftover for web requests.
    # Clamp it to 1 if the configured max threads is 2 or less for whatever reason.
    thread_count = [(ENV.fetch('RAILS_MAX_THREADS',5).to_i - 2),1].max

    games_progress_bar = ProgressBar.create(
      total: games.count,format: "\e[0;32m%c/%C |%b>%i| %e\e[0m"
    )

    # Disable logging in production to prevent log spam.
    Rails.logger.level = 2 if Rails.env.production?

    Parallel.each(games,in_threads: thread_count) do |game|
      ActiveRecord::Base.connection_pool.with_connection do
        begin
          [:small,:medium,:large].each do |size|
            game.sized_cover(size).process
          end
        # Rescue MiniMagick errors if they occur so that they don't block the
        # task from continuing.
        rescue MiniMagick::Error => e
          games_progress_bar.log "ERROR: #{e.message}"
          games_progress_bar.log "Failed on game ID: #{game.id}"
        end
        games_progress_bar.increment
      end
    end

    games_progress_bar.finish unless games_progress_bar.finished?

    users = User.joins(:avatar_attachment)
    # Only attempt to create variants if the avatar is able to have variants.
    users = users.filter { |user| user.avatar.variable? }
    puts 'Creating user avatar variants...'

    users_progress_bar = ProgressBar.create(
      total: users.count,format: "\e[0;32m%c/%C |%b>%i| %e\e[0m"
    )

    Parallel.each(users,in_threads: thread_count) do |user|
      ActiveRecord::Base.connection_pool.with_connection do
        begin
          [:small,:large].each do |size|
            user.sized_avatar(size).process
          end
        # Rescue MiniMagick errors if they occur so that they don't block the
        # task from continuing.
        rescue MiniMagick::Error => e
          users_progress_bar.log "ERROR: #{e.message}"
          users_progress_bar.log "Failed on user ID: #{user.id}"
        end
        users_progress_bar.increment
      end
    end

    users_progress_bar.finish unless users_progress_bar.finished?
  end
end

这就是 sized_cover 中的 game.rb：

def sized_cover(size)
  width,height = COVER_SIZES[size]
  cover&.variant(
    resize_to_limit: [width,height]
  )
end

sized_avatar 几乎是一样的。