Store Files In Background with Carrierwave and SidekiqOctober 08, 2014
When allowing users to upload large files like images and videos it's important to process and store the files using a background process. If you don't, that upload process will tie up the App server and prevent other users from accessing your website. And it's very easy using Carrierwave with the help of the carrierwave_backgrounder gem. This post assumes you are familiar with Carrierwave and have the uploaders set up already, but I wanted to highlight the steps I used to get it working because they were not all incredibly obvious (to me at least).
To start out you need to install Redis, which is the database that Sidekiq uses. On OSX I recommend using Homebrew:
brew install redis
Then you add the sidekiq and carrierwave_backgrounder gems to your Gemfile:
Next install carrierwave-backgrounder by running the following command:
rails g carrierwave_backgrounder:install
Then navigate to config/initializers/carrierwave_backgrounder.rb which was created by the previous install script and uncomment the line:
c.backend :sidekiq, queue: :carrierwave
In your model add store_in_background below the line that mounts your uploader:
mount_uploader :file, VideoUploader
Add a column to the model you want to background which will store the temp file location:
add_column :videos, :file_tmp, :string
and obviously rake db:migrate to implement that migration
Add the following line to your carrierwave uploader file inside of the uploader class:
Next you have to start Redis (if it's not currently running) and Sidekiq:
bundle exec sidekiq -q carrierwave
The '-q carrierwave' specifies the queue name and is important because carrierwave-backgrounder automatically uses 'carrierwave' as it's queue name (though you can change that above in config/initializers/carrierwave_backgrounder.rb) and sidekiq, by default, looks for a queue named 'default'.
Add the following to your config/routes.rb file:
mount Sidekiq::Web, at: '/sidekiq'which will allow you to visit '/sidekiq' and view the progress of the background processes.
That's it! Now your files will be processed and stored in the background to wherever you specified in your carrierwave uploader file (including external storage like S3). This is my setup in the development environment, and I'm currently playing around with the best way to implement it in production. I'll post an update soon.