Store Files In Background with Carrierwave and SidekiqOctober 08, 2014

When allowing users to upload large files like images and videos it's important to process and store the files using a background process. If you don't, that upload process will tie up the App server and prevent other users from accessing your website. And it's very easy using Carrierwave with the help of the carrierwave_backgrounder gem. This post assumes you are familiar with Carrierwave and have the uploaders set up already, but I wanted to highlight the steps I used to get it working because they were not all incredibly obvious (to me at least).

To start out you need to install Redis, which is the database that Sidekiq uses. On OSX I recommend using Homebrew:

brew install redis

Then you add the sidekiq and carrierwave_backgrounder gems to your Gemfile:

gem 'sidekiq'

gem 'carrierwave_backgrounder'

And

bundle install

Next install carrierwave-backgrounder by running the following command:

rails g carrierwave_backgrounder:install

Then navigate to config/initializers/carrierwave_backgrounder.rb which was created by the previous install script and uncomment the line:

c.backend :sidekiq, queue: :carrierwave

In your model add store_in_background below the line that mounts your uploader:

mount_uploader :file, VideoUploader

store_in_background :file

Add a column to the model you want to background which will store the temp file location:

add_column :videos, :file_tmp, :string

and obviously rake db:migrate to implement that migration

Add the following line to your carrierwave uploader file inside of the uploader class:

include ::CarrierWave::Backgrounder::Delay

Next you have to start Redis (if it's not currently running) and Sidekiq:

redis-server /usr/local/etc/redis.conf

bundle exec sidekiq -q carrierwave

The '-q carrierwave' specifies the queue name and is important because carrierwave-backgrounder automatically uses 'carrierwave' as it's queue name (though you can change that above in config/initializers/carrierwave_backgrounder.rb) and sidekiq, by default, looks for a queue named 'default'.

Add the following to your config/routes.rb file:

mount Sidekiq::Web, at: '/sidekiq'

which will allow you to visit '/sidekiq' and view the progress of the background processes.

That's it! Now your files will be processed and stored in the background to wherever you specified in your carrierwave uploader file (including external storage like S3). This is my setup in the development environment, and I'm currently playing around with the best way to implement it in production. I'll post an update soon.