I've been on a journey to become a full stack web developer, and I'm loving every minute of it. This site will help to share my experiences, things I've learned, projects I'm working on and other topics that interest me. I like to include an image with each post to help break up the text and make the site more navigable (and aesthetically pleasing). The images may be relevant to the post in some way or may just be something that's cool/beautiful/inspiring to me at that moment. I hope somewhere along the way some content on this site can be of service to another developer tackling similar programming hurdles, and maybe make their time of it a bit easier. Thank you so much for visiting.
I started scratching the surface of coding initially by creating some simple e-commerce stores. If you've ever had an online store of any kind you know that there are a lot of mundane tasks that are ripe for automation. Now that I'm finally starting to dive into real programming I'm excited to revisit some of these boring tasks that I spent many monotonous hours with. I wanted to start with some screen scraping with the help of Mechanize and Nokogiri, which will ultimately be used to create products in a Spree shopping cart (in another post).
When navigating with Mechanize it's easy if links all have logical and plentiful classes, you can just use links_with(:dom_class => 'foo'). But that never seems to be the case for me and when classes aren't available on the links directly, to identify the link you want to target you will need to use Mechanize's Nokogiri parser:
page.parser.css('.fooclass a')
page = Mechanize::Page::Link.new(noko_obj, agent, page).click
Here's how you'd use it in a basic scraping loop that clicks on each link, visits the page, and displays the page title (obviously you can use Nokogiri to get whatever information you want once you're on the page):
require 'rubygems'
require 'mechanize'
require 'nokogiri'
agent = Mechanize.new
page = agent.get('http://foobar.com')
page.parser.css('.fooclass a').each do |link|
page = Mechanize::Page::Link.new(link, agent, page).click
puts page.title
end
With the ability to move seamlessly between Mechanize and Nokogiri you can single out elements with complicated identifiers and navigate through them with ease, creating complicated scraping loops that get you all the data you need. Nowhere is safe from your scraping endeavors!
I highly recommend Ryan Bates' Railscasts on Nokogiri and Mechanize to get started with the tools, and SelectorGadget to identify DOM elements without obvious class selectors (which Ryan mentions in his videos).
For me the next step was turning this script into a Rake task and using this knowledge to scrape product information and automatically generate Spree Commerce products to avoid the time and hassle of doing this by hand.
One problem I ran into creating my first site with the Spree shopping cart is that there is no way in the Spree admin interface to move a taxon with all of its contained products to a new parent taxonomy or taxon. Luckily this is very easy to do in the Rails console. First find the taxon object that you want to move:
$ taxon = Spree::Taxon.find_by_name("name")
$ parent_taxon = Spree::Taxon.find_by_name("parent_taxon_name")
$ taxon.parent_id = parent_taxon.id
$ child_taxonomy_id = parent_taxon.taxonomy_id
$ taxon.taxonomy_id = child_taxonomy_id
Hope this helps someone.
I've been teaching myself to program for about 6 months now, and I just figured I'd mention my thoughts on what has been helpful, difficult, useful, etc. so far from a big picture perspective.
I started out just wanting to learn Rails to make cool web apps. I hadn't done programming of any kind (other than tinkering with HTML and CSS) so it was a scary decision to finally commit to learning a real programming language (Ruby). I tried a lot of the online interactive tutorials like codeschool, and while they were great as an introduction I would highly recommend a different approach.
The truth is that while initially I just wanted to learn Rails, I quickly ended up needing to understand more about Ruby to do the things I wanted, and that led to needing to understand more about computer science fundamentals. So my first piece of advice is:
MIT has something called Open CourseWare where they post lecture videos up for an entire course in addition to all of the assignments and exams. The professor does a fantastic job of explaining computer science fundamental concepts like assignment and recursion.
If you're just starting out (like me) learning to program can be pretty overwhelming, there's just a never-ending amount of things to learn (but that's part of what makes it so fun!). And if you're learning to program web apps there are so many things to keep track of. If you want to effectively program in Rails you need to learn Ruby, the Rails conventions, HTML, CSS, and Javascript, but then you also need to learn testing frameworks, a bit of SQL and some Git (or other version management), have an understanding of HTTP, know how to navigate/interact with the console (OSX and Linux) not to mention things like SASS and ERB and any GUI you may choose to use (Twitter Bootstrap, etc.). Sure some of these things are pretty simple, but it all adds up to so many things to learn and remember that it can make your head spin! So, my second piece of advice is:
It may seem like progress is slow, but they're all so interconnected that it's tough to pull them apart. Just keep working at expanding your knowledge of each of them as you need them and it'll all slowly start to fit together.
Which brings me to my next piece of advice:
When I first started out I was spending all of my time reading books, watching lectures, doing tutorials and that's understandable because you need a certain amount of knowledge initially to be able to do anything. But all of these things were very abstract and conceptual. I learned the most, by far, when I decided I wanted to build my first web app and started applying them. Every day I would just decide I wanted to complete a specific task and keep working at it until it was done. Sometimes that meant spending hours on Stack Overflow or Google. Then after a while I was spending so much of my time working on the app that I wasn't really learning as much as I needed conceptually or improving the quality of my programming. Now that I understand some of the basics and am working on projects I'm looking for ways to improve the quality of my code and since I don't have many friends that program that means spending time on Github looking at well-coded projects. Like anything else, choosing how to allocate your time is a balancing act.
The next piece of advice is a big one for me:
To be honest this is what kept me from diving into programming a long time ago, it just seemed too difficult, too intimidating. But now that I'm committed to learning it, I still have to avoid psyching myself out. In my experience this is something that people do a lot in their everyday lives and it's a very limiting process. It's easy to look at things like regular expressions or testing languages and say, "I'll learn that later" or even worse "I just don't use those," but I've started a process where every time I experience myself thinking that way, I make it a point to spend time learning that thing I'm intimidated by. The more I've forced myself to do this the more it has become habit, which has served me in my life in general as well.
This will be a list (that I will hopefully update frequently) of specific issues I've run into/things I want to remember so I don't have to go digging around for how to solve them again in the future.
config.assets.precompile += %w(*.png *.jpg *.jpeg *.gif)
If you have the choice put them in /app or /public. Remember this for Spree where the CSS files have to be in /vendor but any images you add do not. Then make sure to follow the advice below on referencing images.
background: url(<%= asset_path 'rails.png' %>");
. ~/.bashrcto reload the .bashrc file in your current shell.
"foo bar".match(/(foo)\s(bar)/)[1]
=> "foo"
page.parser.at_css('span[title="description"]')and it would locate the first <span> element with a title attribute that has the value "description"
!'Foo Bar'.include?('Baz')Very obvious to most people I know, but I just wanted to note it down as a reference.
rails g model User
class User < ActiveRecord::Base
end
rails g controller Users
class UsersController < ApplicationController
end
rails g resource Person name:string
rails g migration AddPartNumberToProducts
git remote add heroku_production git@heroku.com:project-production.git
git remote add heroku_test git@heroku.com:project-test.git
heroku pgbackups:capture --app project-production
curl -o staging.dump `heroku pgbackups:url --app project-test`
pg_restore -O -d database_name file.dump
render partial: 'posts/post', object: @user_postwill assign the @user_post object to the variable 'post'. A more verbose alternative is specifying local variables with:
render partial: 'posts/post', :locals => {:user_posting => @user_post}which allows you to define things more explicitly and include more than one variable.
accepts_nested_attributes_for :product
heroku run bash
view_context.method
redis-cli monitor - A debugging command that streams back every command processed by the Redis server.
info keyspace - lists keyspaces, a concept somewhat similar to databases.
keys * - lists all keys in current keyspace
to access keys you must know the type of the value which you can get with:
type key
you can then use the correct get method depending on the type:
for "string": get key
for "hash": hgetall key
for "list": lrange key 0 10 - the last two ints represent the index range (exclusive)
for "set": smembers key
for "zset": zrange key 0 10 - the last two ints represent the index range (inclusive)
ufw show addedwill show current rules prior to enabling
lsof -n | sed -E 's/^[^ ]+[ ]+([^ ]+).*$/\1/' | uniq -c | sort | tailwhich lists the PID on the right and the number of files it has open on the left. Then you can use
ps -p 1234 -o comm=replacing 1234 with your PID to get info about the program that's causing the issue
Country.where('name LIKE :search', search: '%United%')
find /path/to/folder -name '*query*'
Previously I mentioned a useful tip that has proven essential for me in navigating successfully with mechanize and Nokogiri. Now I want to share another useful thing I've learned.
When a website is structured without uniquely identifiable structural elements or CSS attributes (often this can happen when a lot of content is added within one html tag without classes or IDs), you may need to look to the content of the tag to find what you need. For example you may have a series of unidentifiable <p> tags that look like:
<p>Name: Foo Bar</p>
<p>Description: A delicious energy bar made of the highest quality foo.</p>
<p>Ingredients: foo, sugar, water</p>
agent = Mechanize.new
page = agent.get('http://foobar.com')
ingredients = page.parser.at_css('p:contains("Ingredients")')
ingredients = page.parser.at_css('p:contains("Ingredients")').text[13..-1]
ingredients = page.parser.at_css('p:contains("Ingredients")').text.match(/:\s(.+)/)[1]The (.+) in this case will capture all characters in the <p> tag after the semicolon and space ":\s" until it hits a line break character (which it won't in this situation). The [1] after .match will return the first capture of the regex, in other words all of the characters captured by (.+), which will be all of the ingredients.
When allowing users to upload large files like images and videos it's important to process and store the files using a background process. If you don't, that upload process will tie up the App server and prevent other users from accessing your website. And it's very easy using Carrierwave with the help of the carrierwave_backgrounder gem. This post assumes you are familiar with Carrierwave and have the uploaders set up already, but I wanted to highlight the steps I used to get it working because they were not all incredibly obvious (to me at least).
To start out you need to install Redis, which is the database that Sidekiq uses. On OSX I recommend using Homebrew:
brew install redis
Then you add the sidekiq and carrierwave_backgrounder gems to your Gemfile:
gem 'sidekiq'And
gem 'carrierwave_backgrounder'
bundle install
Next install carrierwave-backgrounder by running the following command:
rails g carrierwave_backgrounder:install
Then navigate to config/initializers/carrierwave_backgrounder.rb which was created by the previous install script and uncomment the line:
c.backend :sidekiq, queue: :carrierwave
In your model add store_in_background below the line that mounts your uploader:
mount_uploader :file, VideoUploader
store_in_background :file
Add a column to the model you want to background which will store the temp file location:
add_column :videos, :file_tmp, :string
and obviously rake db:migrate to implement that migration
Add the following line to your carrierwave uploader file inside of the uploader class:
include ::CarrierWave::Backgrounder::Delay
Next you have to start Redis (if it's not currently running) and Sidekiq:
redis-server /usr/local/etc/redis.conf
bundle exec sidekiq -q carrierwave
The '-q carrierwave' specifies the queue name and is important because carrierwave-backgrounder automatically uses 'carrierwave' as it's queue name (though you can change that above in config/initializers/carrierwave_backgrounder.rb) and sidekiq, by default, looks for a queue named 'default'.
Add the following to your config/routes.rb file:
mount Sidekiq::Web, at: '/sidekiq'which will allow you to visit '/sidekiq' and view the progress of the background processes.
That's it! Now your files will be processed and stored in the background to wherever you specified in your carrierwave uploader file (including external storage like S3). This is my setup in the development environment, and I'm currently playing around with the best way to implement it in production. I'll post an update soon.
I recently acquired a simple stereoscopic phone headset, which have been getting pretty popular lately with things like Google Cardboard and the Duravis Dive. You pop in your phone, the headset simply splits your vision and the phone presents a separate image to each eye giving the illusion of multidimensionality. Once I started playing with it though I realized it was pretty hard to find a list of iPhone apps that support the headset so as I explored the App Store I figured I'd note down all of the ones I've found and tried with my notes about them:
COSMIC ERROR - FREE:
Definitely the coolest VR game I came across on the app store. You direct a spaceship with the movement of your head to collect points and avoid asteroids as they shoot past you. Certainly worth getting and one of the few apps with any replay value.
Homido Player - FREE:
This app offers an internet browser that you can use with your Stereoscopic Goggles and it actually works pretty well. Open the app and select Browser. If you shake the phone/goggles up and down it will give you a crosshair that you can use to interact with the website (click on links/photos/etc.), just hold the crosshair steady on whatever you want to click on for about 2-3 secs. If you bring it to the top of the browser page it will scroll up and vice versa for down and there are some navigation buttons at the very bottom and top of the screen. I was really pleasantly surprised and it actually does give a slight depth to some websites that created a really unique browsing experience. Also keep in mind that if you need to touch the screen to type a web address/etc. only the left half of the screen works (they tell you this but it was easy to forget).
Casa MaracanĂ£ - FREE (you can search Casa Maracana without the tilde and it will show up in the App Store):
I really enjoyed this one, it was the best house/area exploration app. It's a simple tour around a beautiful house and it moves at a slow pleasant pace that won't make you dizzy like some other VR walking apps. Doors open for you as you approach them and you won't find yourself walking through walls like a lot of the other apps. Look all the way down at your feet to start or stop walking. At first it might seem like there's no way into the house but look around a bit and you'll find your way in.
HOLO Virtual Reality - FREE:
This app has 3 modes: 3D, Video, and Live.
In Video mode you're standing in the middle of a crowded square and there are real people passing all around you and you can turn 360 degrees.
In 3D mode you're walking around a strange virtual city with an inception-like quality, some strange faceless beings and some other interesting touches that I'll leave for you to discover. Apparently there's some way to jump because it happened to me accidentally a couple of times. While trying to figure out how I slammed my knee into my desk and almost knocked over my monitor.
The Live mode is a really cool idea, you record video stereoscopically with the camera taking a separate frame for each eye. The objects that pop out in these VR applications do so because they are shifted differently, so I was curious how it would determine those objects in this video app. It doesn't really succeed in that but there are glorious moments when it all comes together for a second or two.
Overall, with it's 3 distinct applications of the technology (but mostly for the very unique 3D mode) this app was worth checking out.
Volvo Reality - FREE:
A really simple app that puts you in the driver's seat as a Volvo XC90 drives around in nature. It's a promotional tool certainly, but I'd say a pretty clever and effective one. It's a strangely soothing experience being passively driven around through beautiful natural scenery. It's not particularly long, but give it a quick watch.
Visit VR - FREE:
This one is very missable. It's just some photos with very little dimensionality or effect.
Appartement - FREE:
A missable house exploration app. You walk through walls a lot, none of the stairs work. etc.
Timpaan Verandawoning - FREE:
Another missable house exploration app for all the same reasons.
Refugio3D - FREE:
A space station exploration app that looks cool but the app was stuck in portrait mode and there was no way to get it to the correct landscape orientation. A lot of people were complaining about it in the App Store comments but it still wasn't fixed. I'll update if it ever gets corrected.
London VR - FREE:
I pretty simple app that takes you to beautiful stationary positions in London where you can look around in a freeze framed 360 degree view. To enable dual view VR click the settings icon in the upper right corner of the main menu.
Beenoculus Flight Simulator - FREE:
This looked really good, but it was kinda disappointing because it seemed like there was potentially a way to actually fly the plane. It shows a control binding menu at the beginning but only allows you to bind them to a joystick, so I don't know.
Beenoculus Tuscany - FREE:
This company cracks me up. This app is upside-down and there doesn't seem to be any way to right it. The place looks beautiful but unless you like walking around on your hands, this app probably won't be particularly enjoyable to you.
Beenoculus Orinthalian - FREE:
Alright finally one that works. It's simple, but a fun little hut and surrounding area to explore.
Beenoculus SmartCity - FREE:
You stand on top of a pillar and look around at a city. If you look at one part for a couple of seconds it will zoom in on that part a little bit. If you look down at your feet it will zoom back out. Not particularly exciting.
DINOTOWN - FREE:
This one was pretty comical, but worth a watch. It's a small town filled with awkwardly moving 3D dinosaurs. It's not a free roam app, it takes you on a ride through it.
Microgaming Mobile VR - FREE:
You stand in a dark world and spin a roulette wheel. Not my idea of a good time.
Defend Santa's Grotto - FREE:
Little snowmen are coming to steal from Santa's Grotto and you must stop them with snowballs. The game is fun and offers dual view VR but there's no mechanism for throwing other than touching the screen so you can't actually play it on a headset. Quite frustrating.
FastHit VR $.99
VR BasketBall $.99
Pakyow is a fascinating realtime Ruby framework that uses data binding and websockets to keep views in sync with the data they represent. I haven't worked with other realtime Ruby frameworks (like Volt) yet so I can't compare, but my impression is that many of them re-render the page on the server and push it to the client (browser) whereas Pakyow just makes all of the changes server-side to alter objects that are bound to front end elements, updating the state of those front-end elements in realtime using websockets. This has the benefit of keeping the DOM intact as opposed to re-rendering it. Also you're not writing ruby on the front end (via a JavaScript compiler like Opal), and while I haven't had the chance to benchmark it yet, my anecdotal impression is that the performance improvements of Pakyow's approach are substantial.
Pakyow is also designed to degrade gracefully so if websockets stop working correctly or even JavaScript altogether your web app will still function, changes just won't happen in realtime.
I'm extremely excited about the possibilities of the framework. I can tell that I'm going to love the realtime functionality and it seems like the interface is fairly elegant as well but my impressions is that it will be a somewhat steep learning curve. The docs are good and cover a lot of functionality, but in my personal opinion they don't seem to be organized in a logical way to help ramp up a beginner and get them familiar with the framework and interfaces. In this post I'll be documenting things I find helpful or interesting as I dive in and learn Pakyow.
Notes
Access Mutables you've created with the data(:scope) method
The process of training Tessaract OCR on Mac OSX can be pretty confusing. The Tessaract documentation isn't great and most of the existing blog posts and information online refer to training Tesseract on Linux or Windows. My hope is to document my process from start to finish to give others a more complete guidance as they navigate through the process.
Start by installing Tesseract using homebrew with training tools. Until a recent contribution by Ryan Baumann that used to involve building from source, but thanks to him we can just use:
brew install --with-training-tools tesseractThere can be a few gotchas (particularly with font locations --fonts_dir) so be sure to refer to his post for more information. (Article)
Next make sure you've got the latest version of Java SE Development Kit installed on your mac. (Latest version at time of writing)
Then download jTessBoxEditor which is a training software we'll be using. Download the latest version with the format jTessBoxEditor-version.zip. (jTessBoxEditor Download)
Then start jTessBoxEditor by running the following command in the unzipped jTessBoxEditor directory:
java -Xms4096m -Xmx4096m -jar jTessBoxEditor.jar
I needed to integrate Action Cable into a Rails 4.2 app with Devise and I figured I'd document what worked for me. Strangely, none of the example repos on Github worked out of the box but following this tutorial with a few changes was the best place to start and creating a Warden hook as specified here proved to be the easiest way to get Devise working as well. Here are the steps I took and some of the pitfalls I encountered. We'll be adding an arbitrary chat component to a Rails 4 app where a user can only talk to themselves. Admittedly not a super useful feature but it will demonstrate how a channel can be scoped to a specific user, which is where I experienced the most issues.
Start by adding Action Cable and Puma (if you aren't already using it) to your Gemfile. Now that Action Cable has been integrated into Rails you have to point your Gemfile to the 'archive' branch:
gem 'actioncable', github: 'rails/actioncable', branch: 'archive'
gem 'puma'
resources :messages, only: [:index, :create]
# app/config/initializers/warden_hooks.rb
Warden::Manager.after_set_user do |user,auth,opts|
scope = opts[:scope]
auth.cookies.signed["#{scope}.id"] = user.id
auth.cookies.signed["#{scope}.expires_at"] = 30.minutes.from_now
endWarden::Manager.before_logout do |user, auth, opts|
scope = opts[:scope]
auth.cookies.signed["#{scope}.id"] = nil
auth.cookies.signed["#{scope}.expires_at"] = nil
end
This uses hooks in the Devise authentication lifecycle to set and expire signed cookies.
Next add a messages controller:
class MessagesController < ApplicationController
before_filter :authenticate_user!
protect_from_forgery :except => :createdef create
ActionCable.server.broadcast "messages_#{current_user.id}",
message: params[:message][:body],
username: cookies.signed['user.id']
head :ok
end
end
<h2>Messages</h2>
<br/><br/>
<div id='messages'></div>
<br/><br/>
<%= form_for :message, url: messages_path, remote: true, id: 'messages-form' do |f| %>
<%= f.label :body, 'Enter a message:' %><br/>
<%= f.text_field :body %><br/>
<%= f.submit 'Send message' %>
<% end %>
# app/channels/application_cable/connection.rb
module ApplicationCable
class Connection < ActionCable::Connection::Base
identified_by :current_userdef connect
self.current_user = find_verified_user
logger.add_tags 'ActionCable', current_user.name
endprotected
def find_verified_user
verified_user = User.find_by(id: cookies.signed['user.id'])
if verified_user && cookies.signed['user.expires_at'] > Time.now
verified_user
else
reject_unauthorized_connection
end
end
end
end# app/channels/application_cable/channel.rb
module ApplicationCable
class Channel < ActionCable::Channel::Base
end
end
Action Cable relies on Redis' PubSub feature so you need to have Redis installed and you need to add a config file (config/redis/cable.yml):
local: &local
:url: redis://localhost:6379
:host: localhost
:port: 6379
:timeout: 1
:inline: true
development: *local
test: *local
Though apparently you can integrate Action Cable directly into your existing app process, we are setting it up as a separate server so we need to create a bin/cable executable and a rackup file:
# /bin/cable
bundle exec puma -p 28080 cable/config.ru
chmod 711 bin/cable
# cable/config.ru
require ::File.expand_path('../../config/environment', __FILE__)
Rails.application.eager_load!require 'action_cable/process/logging'
run ActionCable.server
# app/channels/messages_channel.rb
class MessagesChannel < ApplicationCable::Channel
def subscribed
stream_from "messages_#{current_user.id}"
end
end
#= require cable@App = {}
App.cable = Cable.createConsumer("ws://localhost:28080")
App.messages = App.cable.subscriptions.create 'MessagesChannel',
received: (data) ->
$('#messages').append @renderMessage(data)renderMessage: (data) ->
"<p><b>[#{data.username}]:</b> #{data.message}</p>"
bin/cableand in the second fire up your rails server:
bundle exec rails sOpen up two browser windows, log in on both of them and visit localhost:3000/messages. When you send a message you should see your id and your message come up simultaneously in both browser windows. Hooray!
$> redis-cli
$> monitor
Again, credit and many thanks to Nithin Bekal for his post that was the basis for this one and the only tutorial that worked for me by default. Also thanks to Greg Molnar for his post on integrating Devise.
I recently found myself in a situation where I needed to use Bootstrap tab functionality but the tab menu needed to be split into sections with markup between the sections. In the Bootstrap guides and all of the examples I found, people used a list (ordered or unordered) and list items to create the tab menu, something like:
<ul class="nav nav-tabs">
<li class="active"><a href="#home" data-toggle="tab">Home</a></li>
<li><a href="#profile" data-toggle="tab">Profile</a></li>
</ul>
<nav>
<div class="nav-item active"><a href="#home" data-toggle="tab">Home</a></div>
<div class="nav-item"><a href="#profile" data-toggle="tab">Profile</a></div>
</ul>
Now you can easily insert markup between the nav-items to your hearts content:
<nav class="row">
<div class="col-md-6">
<div class="nav-item active"><a href="#home" data-toggle="tab">Home</a></div>
</div>
<div class="col-md-6">
<div class="nav-item"><a href="#profile" data-toggle="tab">Profile</a></div>
</div>
</ul>