Possible Email Ruby Gem

PossibleEmail Gem on GitHub PossibleEmail on RubyGems

This is a continuation of my projects relating to using the undocumented Rapportive API. My previous project, Find Anyone’s email: A Ruby script, was a popular project that spurred an issue on Github requesting the script to be written as a gem. After some work, here is the final product.

There were a few motivations in making this gem. When I first released the find-any-email script, I always thought of it as a proof-of-concept instead of the end all be all. Not only did it feel like a hack, but I thought about how easy and interesting it would be to integrate the functionality into a web app if the script was converted into a gem. Another motivation I had going into this project was to improve on my RSpec specification writing. I recently finished reading through the RSpec Book, and I wanted to practice writing specs for this project. If you actually take a look at the gem’s source code, then you might notice there are 75 tests for the gem. Those 75 tests were the result of going through TDD process of Red-Green-Refactor numerous times.

I’m interested to see how this gem would be used in other people’s projects. I was actually considering of creating a sort of SaaS product around the gem or an iPhone app. It seems that these types of email-mining things are of value to some people specifically sales and business people. I actually have something up that I built a few months ago, DirectContact, but I put it on the shelf at the moment. The ideas still seems to have value for people, so I might pick it back up after finishing another project. It gets a bit old thinking about a fairly simple idea after you’ve been thinking about it for awhile.

I would like to tip my hat to Jordan Wright for being a big inspiration for this gem after he wrote his python library Rapportive

Railscast 032 Time in Text Field

Project GitHub Repo

StringifyTime Gem GitHub Repo

The Heroku App

This episode was essentially repeated in Railscast 016 Virtual Attributes. Despite it being a repeat, I decided to create a project for this Railscast because the following episode, 033, used it to make a Rails Plugin. Rather than make the plugin, I wanted to try creating my second RubyGem based on this plugin.

Although I was not aware of this while making the gem, there was another person who had the same idea. His gem is called jakewendt-stringify_time, but it hasn’t been updated four years. The gem I created integrates with the changes made to the Ruby on Rails framework since that gem’s creation.

Because this would be my second gem I have created, I thought I could just model this gem similarly to my first gem body_id. I created the basic gem files using bundle gem stringify_time command. Then I edited the .gemspec file to put in the basic information needed. The part I got confused about was at the part I was going to use railtie. The Railscast simply extended ActiveRecord using the module created in the plugin. Instead of following my previous gem, I decided to try how it was made in the video. I copied the StingifyTime module from the episode and extended ActiveRecord just like it was shown in the episode. I ran rake release and had version 0.0.1 on RubyGems.org. That’s when I created the previous project from Episode 032 to try the gem out. The bad part was, it didn’t work.

After some searching around I noticed other gems did not use extend method on ActiveRecord directly, but they used the .send :include, StringifyTime method to call the include method on a module. So I tried to exchange

ActiveRecord::Base.extend StringifyTime

with

ActiveRecord::Base.send :include, StringifyTime

The problem was I blindly tried to copy and try this method without fully understanding why I was using include instead of extend. Include is used on an instance of a class to add methods, while extend is used to add methods to the class itself. At the same time, I searched my gem on RubyGems.org and discovered the Jake Wendt’s gem based on the same episode. I looked at his GitHub Repo and saw he had an init.rb and rails/init.rb. “Could this solve it?”, I thought. I applied the changes, and the stringify_time method was being include in my rails app.

Metaprogramming

What is so interesting about creating this stringify_time method is that you are writing a program in order to have it write another program. That’s the concept of metaprogramming at least. You could have defined each getter and setter method manually, but that isn’t practical when you have numerous attributes you need to do the same thing on. I have used metaprogramming when I made a seed.rb file to fill up the database with records. These methods saves a lot of time if made correctly.

Unresolved

When I went to edit the date on the form, the date was not updating. This is still a problem I need to solve, and I have opened a question on StackOverflow to try to resolve this issue. While I wait for an answer I’ll keep moving to another Railscast.

Update

Someone on StackOverflow suggested I try using the generated method via the Rails console. I tried to use the rails console, and the due_at attribute was updated. I thought why was I able to update the attribute in the console, but not via the form. The answer was because I was not whitelisting the :due_at_string param in the task_params definition. This is to protect from mass assignment. So the data from the form was never reaching the model because it was being prevented by the controller. A silly mistake to overlook.

Railscast – 002 Dynamic find_by Method

The GitHub Repo

I was arguing if this episode was even worth creating a project for. Decided to create since it would be so quick to make.

The episode talks about how you can replace:

class TaskController < ApplicationController
  def incomplete
    @tasks = Task.find(:all, :conditions => ['complete = ?', false])
  end

  def last_incomplete 
    @task = Task.find(:first, :conditions => ['complete =?', false], :order => 'created_at DESC')
  end
end

With:

class TaskController < ApplicationController
  def incomplete
    @tasks = Task.find_all_by_complete(false)
  end

  def last_incomplete
    @task = Task.find_by_complete(false, :order => 'created_at DESC')
  end
end

Attention should be drawn to the find_all_by_complete method used. This is a shortcut for the first find methods with all of the options passed in. The word that follows the by is the column that is in the Tasks table. You can then pass in the value of the records for that column you want returned. If you want only a single record that matched the conditions, then you would omit the all from find_all_by_complete making it find_by_complete. If you were finding a record by its name, then you doing this:

Task.find_by_name('bob')

Railscast – 016-Virtual-Attributes

The GitHub Repo
The Heroku app

I found this episode useful. It showed me that a model’s attribute does not have to be directly on one of the table’s columns. Not only did I learn about virtual attributes, but I also improved on associations, validations, and callbacks.

The virtual attributes were easy enough to understand. You create a setter and getter method named with the desired virtual attribute name. In the episode’s case it was price_in_cents sort of aliased to price_in_dollars. Within the method you could change the value of the virtual attribute to match whatever is in the database.

def price_in_dollars
  price_in_cents.to_d / 100 if price_in_cents
end

def price_in_dollars=(dollars)
  self.price_in_cents = dollars.to_d * 100 if dollars.present?
end 

Strftime

This is the first time I used the :strftime method, but it is not too dissimilar to how I have formatted dates within the terminal.

Rails 3 to Rails 4

One thing to note, these videos date back sometime before the current Ruby on Rails 4.0 release, thus I must convert some of the techniques shown in the videos into whatever they translate to in the new version of the framework. An example of this is the use of attr_accessible. Attr_accessible has been incorporated in the the strong parameters update in Rails 4. So instead of

attr_accessible :name, :price_in_dollars, :released_at_text, :category_id, :new_category, :tag_names

You would do the following in the ProductsController

def product_params
  params.require(:product).permit(:name, :price_in_dollars, :released_at_text, :category_id, :new_category, :tag_names)
end

Associations

I knew I was a little weak with associations, so when I saw that this project had categories, products, tags, and taggings, I decided to make everything from scratch. I followed along with the schema to create the migrations needed. When creating a has_many and belongs_to relationship you always put the parent id onto the child’s table. Example:

class Product < ActiveRecord::Base
  belongs_to :category
end

class Category < ActiveRecord::Base
  has_many :products
end

The schema would look like this:

 create_table "categories", :force => true do |t|
    t.string   "name"
    t.datetime "created_at", :null => false
    t.datetime "updated_at", :null => false
  end

  create_table "products", :force => true do |t|
    t.string   "name"
    t.integer  "category_id"
    t.datetime "created_at",     :null => false
    t.datetime "updated_at",     :null => false
  end

This relationship is similar to the taggings, tags, and products relationship. The taggings has the foreign key for both the tags and products. The difference is that products has_many tags through the taggings model.

Validations

I liked how Ryan Bates created his own custom validation method with check_released_at_text The validation first check if the released_at_text instance variable is present and the time is nil. If it is, then add to the errors object with a custom error message. If there was an ArgumentError, then again the errors object is appended with the custom error message. Creating a custom validation adds more control than if some of the more generic validation methods like presence or length.

Callbacks

The callbacks created this episode used before_save. The reasoning for this is to actually save the instance variable’s values to the database in the correctly formatted form. In the create_category callback it actually creates a new category record.

  def create_category
    self.category = Category.create!(name: new_category) if new_category.present?
  end

Pluck

The first time I’ve seen the pluck method used. I looked it up and found that tags.pluck(:name) is a shortcut for tags.map(&:name). The goal is to get only a certain attribute from the model.

Those where the noteworthy parts of the episode.

Make all Railscast Projects

I came across a post about someone asking what was the best way to learn Ruby on Rails via Quora. One answer suggested one of the best ways was to follow along with every Railscast project then upload the repo to both GitHub and Heroku. I liked the concept, so I have bought a membership of Railscast and have downloaded all the episodes.

While the goal is to create every project, I will watch every episode and decide if it would be useful to make the project or not. Some episodes are just small tips, so creating a new project just to use that one tip seems inefficient.

I might be repeating what episodes showed, but I found that writing about what I learnt from the episode solidifies what I have learnt. Similar to language learning, you need a balance of input and output to get good at any skill. The input here are the Railscast episodes and the things I look up to finish the apps, and the output are the working app and the blog post.

The first few days of trying this method, I created an app then wrote up the blog post. This takes a lot of context switching between tasks. I am currently trying to go through as many Railscasts in one day, then writing about each the following day.

Change the data type of a column in the database

I accidentally created a column in the database with the datatype of string when it needed to be a integer. I’m on the BART, so I didn’t have internet. Fortunately I have a copy of the Rails repo and its guides. I opened up migrations.md and read to the changing tables section. Saw the following:

change_table :products do |t|
  t.remove :description, :name
  t.string :part_number
  t.index :part_number
  t.rename :upccode, :upc_code
end

I followed this and created the following migration:

change_tabe :products do |t|
  t.remove :price_in_cents
  t.integer :price_in_cents
end

Generate a scaffolded controller after making model

I’m starting to post each time I have to look up something online. I first created a products model, then I migrated the database. I needed a normal scaffolded ProductsController, and I found a StackOverflow question. “rails generate scaffold” when model already exists

I didn’t know there was so many generators until I saw the output of rails generate -h

controller
generator
helper
integration_test
mailer
migration
model
observer
performance_test
plugin
resource
scaffold
scaffold_controller
session_migration
stylesheets

Typed in rails g scaffold_controller product, and it did as it was supposed to do.

Find Anyone’s Email: A Ruby Script

Rapportive is no longer available, so this script no longer works

I just released my Find-Any-Email Ruby Script online a few days ago, and it has been picked up by a few people in the ruby community. In a little over a day, I have 101 stars, 15 forks, and over 3,000 views on the GitHub repo. Its my first time experiencing people using something I made at this scale.

That being said, credit must be given where credit is due. I stand on the shoulder of giants after all. I borrowed heavily from Jordan Wright’s Rapportive.py. I wasn’t too familiar with Python, so I ported it into Ruby. I got the idea for the name permutations from Distilled.net’s post about How To Find (Almost) Anybody’s Email Address. And honorable mention to Eric Wang’s post about how to get a startup job for a ‘guide’ on getting a job and hustling.

I used the manual process of hovering over each email within gmail with the rapportive plugin previously to get my internship in Tokyo. Now that I’m in San Francisco, I thought I’d repeat the process again here. After finding companies I would be interested working for, I searched for the highest person’s email address. Then I sent them a cold email. Not surprisingly some ignored my email, but some answered and were interested in meeting. After hovering over hundreds of email address, I got tired of it.

Then I remembered I knew how to program, so I could programmatically solve my own problem. That’s when I searched for Rapportive API and found Jordan Wright’s blog post. It’s a great read and way to learn how a security engineer works. Because of the similarities between Ruby and Python, I could fairly easily understand the code. Then I started searching for what the Ruby equivalent would be for some methods. Methods like Python’s request() to Ruby’s OpenURI library. I found porting a Python script into Ruby to be good exercise to understand both languages better.

Problems arose when I tried to port everything one-to-one. Some requests to Rapportive’s API were giving 429 and 403 errors. 429: Too many Requests, 403: Forbidden. In order to by pass this, I implemented a begin and rescue block with exponential backoff. Initially I didn’t even know about the concept of Exponential Backoff until I went a meetup downtown and someone suggested that I use it. I’m really grateful for learning about that because it’s so useful when working with APIs! Another note about the 429 and 403 errors, most of the time if these errors comes up, then the email does not have a person associated with them.

The emails were created using the permutate method which is given the arguments of first_name, last_name, and domain. Implementing the permutations and a method to loop each one into the request was the biggest contributions I had to this emailing finding technique. I was going to allow for the middle name argument, but decided against it after running into some problems when its left blank. The permutate method bothers me alittle because I can’t help but feel there is a better way of doing this. I have opened a question on stackoverflow, but I have not received a decent answer. In this part, I’m open to suggestions.

Comments have arisen about the morality of making such a script public. My response is that this technique has already be public for a long time before this script was published. Since this script borrows heavily from Rapportive.py, there is nothing that prevented people from doing this before. Even at a lower level, people aren’t going to email someone without a purpose. Example, Mark Cuban’s email is public, but I’m not going to spam him because it would be a waste of his and my time if I have nothing interesting to email about. It really just gives this technique exposure to Rubyist.

Another point is that this script is admittedly imperfect. Some valid email address are being skipped because Rapportive’s API is returning an error. You could possibly cross reference Rapportive’s API with gravatar’s or any other services’s API. Also every valid email is outputted instead of the one that you’re really looking for. To solve this, you could check that the name in the response matches with the inputted first and last name. I have opened bug issues on GitHub noting these problems. Because of these issues, I use this script as the first pass before doing it manually.

This the first time I pushed something online where I got this attention. It gets me excited to make more things and releasing it to the public.

Questions, comments, or concerns?
Follow me on Twitter¯

Bitcoin donation appreciated
18fZ6muNmBrtENMZhnAjUw8eEsytmY8mZJ

 

My First Ruby Gem: body_id

I created my first Ruby Gem: body_id on GitHub, and I have successfully implemented it my own project. My goals were to see what goes in making a gem and making a process simpler.

The Problem
I first saw the use of @bodyid when Derek Sivers announced that he open sourced all his projects. Which was useful in learning about Sinatra He had @bodyid in his controller, and he called it within the body tag.

When I saw this, I thought it was incredibly useful for styling, so I used it on my own rails project. The problem arose when I was defining my @bodyid to strings like page-index or new-car. Eventually each controller action had a @bodyid.

The Solution
I searched online if anyone already made a gem that did this, and I found these two links. Automatic Body ID in Rails on GitHub and Create a simple Rails 3 text helper Gem by on StackOverflow. The first link showed how you could combine the controller and action name into a single string. I like how he extracted my current process out into a helper, but I saw there were a few unnecessary method calls.

The second link essentially asked how to make the same gem I was planning on making. What I liked about his approach was how he made the id both semantic and similar to RESTful Rails path name. i.e. new_page_path and new_page_view. The answers and the asker’s GitHub repo was key to showing me how to make the actual gem.

There was a problem with these two helpers however. They both did not allow for the user to define their own ID in case they wanted  to diverge from the helper’s convention. That’s where I added a simple || operator with the generated ID and an instance variable defined within the controller.

The final helper:

    def body_id
      controller = controller_name.singularize.underscore
      action = action_name.underscore
      body_id = “#{ controller }-#{ action }”.gsub(/_/, ‘-‘)
      # Controller declared instance variable
      # or helper generated string
      @body_id || body_id
    end

I first removed the controller object in favor to just calling controller_name and action_name directly. This eliminates the need for the gsub() to make ‘controller’ to nothing. Next was the underscore(). I wasn’t sure why I needed this method until I remembered that some controller names and action names were written in CamelCase, so if I wanted to have the generated output look more consistent, then it would be best to leave it in. Finally I combined the two variables into the body_id variable after replacing _underscore with – hyphens to match with CSS conventions. The last line determines if we’ll use the controller declared instance if truthy or the helper’s generated string if @body_id is not defined.

I used bundler to generate the gem files I needed, while I referenced RailsCast #245 New Gem with Bundler and Railtie’s docs One of the things I read online about making a gem was to add some graphics love and make good README. So I got some Illustrator practice in while making the gem’s logo, and I looked at other README for common practices.

Overall I found the process easier than I thought it would be and helpful in understanding what goes into making a simple open source Ruby Gem available.

Fork me on GitHub the4dpatrick / body-id-gem