Quick and dirty Ruby CGI script to collect email addresses

no comments Posted by seth Fri, 14 Sep 2007 10:41:00 GMT

I needed to quickly throw together a CGI script to grab email addresses from a form. If you couldn’t tell, Ruby is my language of choice.

It’s kind of weird not doing some full-blown Rails application, so I actually had to go and look up how to configure Apache to run CGI scripts again.

Here’s the script in all it’s quick and dirty glory.

It expects you to have Ruby, FasterCSV and Ruby Gems installed on your machine.

All it does is collect email addresses to write to an Excel-compatible CSV file, then renders a thank you page (sign_up_thanks.html, you should create one).

#!/usr/bin/env ruby

# This simple CGI ruby script is to collect email
# addresses of fans that might want to sign up for our mailing list.
#
# It's just a CSV file, with a format like so:
# name, email_address, sign_up_date
#

require 'rubygems'
require 'cgi'
require 'fastercsv'

DBFILE = 'emails.csv'
OUTPUT = 'sign_up_thanks.html'

# Parse CGI params
cgi = CGI.new

# Append to CSV file
FasterCSV.open(DBFILE, 'a') do |csv|
  csv << [cgi['name'], cgi['email'], Date.today]
end

# Render thank you page...
cgi.out('text/html') { File.read(OUTPUT) }

Ruby MySpace Friend Adder Script v1.0

no comments Posted by seth Wed, 11 Jul 2007 08:54:00 GMT

A lot of you out there might or might not know that I’m really into producing music.

I like to promote my group on MySpace, like everyone else these days. However, adding friends one by one sucks and is a big waste of time.

There are a lot of pay scripts and programs out there, but who really wants to shell out $60 for such a simple program?

Since I know a little bit about developing software I took a few hours out of the day today to put together a MySpace friend adder script written in Ruby.

How it works

It first prompts you for your MySpace username and password. It then will ask you what you want to search Google for.

You can enter any search term you wish, and it will start harvesting MySpace friend URLs to add as friends to your account.

It stores these friends in a local SQLite database. You can cancel at any time by pressing ctrl-C. When you run the script again you will be asked if you’d like to continue where you left off, or start a new search.

It will begin adding people to your account. When it encounters a CAPTCHA (you know, that little weird image thing) it will open up FireFox for you to type it in.

When you’re done, just come back to the command line and hit return and the script will continue running.

CAUTION

Be sure to only add a couple hundred people to your account per day. Myspace will freak out and possibly shut your account if you’re trying to add 2000 people a day or something insane.

Moderation is good for everything…

Terms of Use

The script is free to use but if you feel like, please donate.

Installation

You will need to have Ruby and Ruby Gems installed.
Follow the directions here if you don’t know how to do that. (Just follow the Ruby and Ruby Gems parts…)

You will also need to install the following Ruby Gems (gem install [gemname])
  • activerecord
  • mechanize
  • hpricot
  • sqlite3-ruby

Configuration

Download the script here and save it to your local disk.

If you edit the file you’ll see there’s a BROWSER_EXE command path. It expects you to be using firefox, on a mac. If you use windows uncomment the second BROWSER_EXE line. I haven’t tested it on windows, and don’t plan to…ever. You all can be my lab rats.

Running the script

Dead easy…open up a command prompt and type:
ruby myspace_adder.rb

Here’s the script…

#!/usr/local/bin/ruby

# MySpace friend adder script.
#
# - Searches Google for a term you give it.
# - Rips friend ID's to a SQLite database in memory.
# - Adds those friends to your MySpace.
#
# !!! THIS SCRIPT IS FREE TO DISTRIBUTE, JUST KEEP THESE COMMENTS INTACT !!!
#
# Work by Subimage.com
# http://sublog.subimage.com/articles/2007/07/11/ruby-myspace-friend-adder-script
#
# REQUIRED GEMS:
#
# - activerecord
# - hpricot
# - mechanize
# - sqlite3-ruby
#

# GLOBALS =====================================================================

# STUFF YOU SHOULD CHANGE

RECORDS_PER_GOOGLE_PAGE = 100
GOOGLE_PAGES = 10
# Should enter these here, otherwise it'll prompt you for it...
MYSPACE_USERNAME = ''
MYSPACE_PASSWORD = ''

# mac
BROWSER_EXE = 'open -a firefox'
# windoze
# BROWSER_EXE = 'c:/progra~1/mozill~1/firefox.exe'

# Maximum sleep time between page reloads in seconds
MAX_SLEEP_TIME = 10

# DON'T CHANGE THIS STUFF

SCRIPT_VERSION = "1.0"
DBFILE = "myspace_adder_db.sqlite"
SEPARATOR = "--------------------------------------------------------------------------------"
# User agents
UA = [
   'Windows IE 6' ,
   'Windows Mozilla',
   'Mac Safari' ,
   'Mac Mozilla' ,
   'Linux Mozilla', 
   'Linux Konqueror' ]

# SETUP CONFIGURATION =========================================================
begin
  require 'rubygems'
  require 'active_record'
  require 'open-uri'
  require 'mechanize'
  require 'fileutils'
rescue
  puts "!!! SETUP ERROR !!!"
  puts "Please make sure you have followed the setup directions at: "
  puts "http://sublog.subimage.com/articles/2007/07/11/ruby-myspace-friend-adder-script"
end

class Friend < ActiveRecord::Base

end

class MySpaceAdder

  # Initializes...
  def initialize
    # Initialize mechanize agent
    @agent = WWW::Mechanize.new
    # Set random user agent
    @agent.user_agent_alias =  UA[rand(UA.size)]
    @myspace_username = MYSPACE_USERNAME
    if @myspace_username.blank?
      printf "Please enter your MySpace email address / username: "
      @myspace_username = gets.chomp
    end
    @myspace_password = MYSPACE_PASSWORD
    if @myspace_password.blank?
      printf "Please enter your MySpace password: "
      @myspace_password = gets.chomp
    end
  end

  # GOOGLE SEARCH
  #
  # Looks at google for phrase given and snags friend IDs
  #
  def do_google_search

    puts SEPARATOR
    puts ""
    puts "Let's search Google to find friends to add..."
    printf "What should we search for: "

    search_string = gets

    query  = "http://www.google.com/search?q=interests+#{search_string}"
    query << "+site:www.myspace.com&num=#{RECORDS_PER_GOOGLE_PAGE}"

    # Open url, get up to 10 pages of links since that's all google gives
    # Each page contains 100 search results...(or whatever you've set the variable to)
    puts "\n\nGrabbing page results..."
    for page_index in 0..GOOGLE_PAGES do
      puts "...getting page #{page_index}"
      query << "&start=#{page_index * RECORDS_PER_GOOGLE_PAGE}"
      page = @agent.get(query)
      # Rip out all the friend names
      page.links.each do |link|
        str_to_look_for = "www.myspace.com/"
        start_pos = link.href.index(str_to_look_for)
        next if !start_pos
        # Offset start slice pos
        start_pos = start_pos+str_to_look_for.size
        next if link.href.index("+")
        next if link.href.index("&")

        username = link.href[start_pos, link.href.length-1]

        if !Friend.find(:first, :conditions => ["username = ?", username])
          Friend.create(
           :username => username,
           :link => link.href
          ) 
        end
      end
      # Sleep between page grabs so google doesn't freak out...
      sleep_time = rand(MAX_SLEEP_TIME)
      puts "...sleeping #{sleep_time} seconds so Google doesn't freak out..."
      next if page_index == GOOGLE_PAGES
      sleep(sleep_time)
    end

    puts SEPARATOR

    puts "Total friend count: #{Friend.count()}"
    puts "Done searching...time to add these suckers"
  end

  # Tries to login to myspace...
  #
  def myspace_login
    # Login to myspace...
    puts "...Trying to login to MySpace..."
    page = @agent.get("http://www.myspace.com/?Mytoken=90f1eeb2-f110-401b-a150-d9053692048f")
    # Fill in the form
    login_form = page.form("theForm")
    login_form.email = @myspace_username
    login_form.password = @myspace_password
    # Click the login button
    page = @agent.submit(login_form)
    if !page.form("theForm")
      puts "...We're logged in!"
    else
      puts "...Please check your login and password. We couldn't log you in."
      exit
    end
  end

  # ADD FRIENDS
  #
  # Adds friends stored in the SQLite database.
  #
  def add_friends

    puts SEPARATOR

    self.myspace_login()

    # Loop through all friends
    Friend.find(:all).each do |f|
      # Navigate to person's page, find the add link, go there...
      puts "\n\nTrying to add: #{f.username}"
      # Remove friend from the db...just because we don't want to
      # do any odd condition checking later.
      f.destroy
      #
      @agent.read_timeout = 30
      begin
        page = @agent.get(f.link)
      rescue Timeout::Error
        "Getting the page is taking too long...skipping."
        next
      rescue
        "General error trying to load page...skipping"
        next
      end

      add_link = nil
      # Mechanize's find method on urls is shitty, so lets do this
      # oldschool
      while add_link == nil do
        page.links.each do |link|
          add_link = link.href if link.href && link.href.index('addfriend')
        end
        # Can't find the add link, move on...
        puts "Add link: #{add_link}"
        break if !add_link
      end
      # Get contents so we can parse...
      page_contents = @agent.get_file(add_link)
      page = @agent.get(add_link)

      # Look for the "confirm add friend button"
        if (page_contents =~ /person.is.already.your.Friend/)
            puts "#{f.username} is already your friend."
            next
        elsif (page_contents =~ /pending.friend.request/)
            puts "Pending add request."
            next
        elsif (page_contents =~ /only.accepts.add.requests.from/)
            puts "Private profile. Moving on."
            next
        elsif (page_contents =~ /from.bands/)
          puts "#{f.username} doesn't accept requests from bands..."
          next
        elsif (page_contents =~ /Invalid.FriendID/i)
          puts "Invalid friend ID, moving on..."
          next
        elsif (page_contents =~ /CAPTCHA/)
        puts "********************"
        puts "* CAPTCHA DETECTED *"
        puts "********************"
            puts add_link

            # I put escaped quotes around the '&' in the next line so that my bash 
            # wouldn't interpret it as a line break and foul up the system command
            system("#{BROWSER_EXE} \"#{add_link}\"")
          puts "Press enter when you're typing it in..."
          gets
          next
        end

        add_form = page.form("addFriend")
        if add_form && add_form.class != String
          @agent.submit(add_form)
        else
          puts "Couldn't find the add form...moving on"
        end

      # Sleep between friend adds grabs so myspace doesn't freak out...
      sleep_time = rand(MAX_SLEEP_TIME)
      puts "...sleeping #{sleep_time} seconds so MySpace doesn't freak out..."
      sleep(sleep_time)

    end
  end
end # / END CLASS

# DO THIS!@
puts "********************************************************************************"
puts "        S U B I M A G E     M Y S P A C E     F R I E N D     A D D E R"
puts "********************************************************************************"

#ActiveRecord::Base.logger = Logger.new(STDERR)
ActiveRecord::Base.colorize_logging = false

ActiveRecord::Base.establish_connection(
  :adapter => "sqlite3",
  :dbfile  => DBFILE
)

adder = MySpaceAdder.new()

if File.size(DBFILE) > 0
  puts "An existing database filled with (#{Friend.count()}) friends has been found."
  printf "Would you like to use it to finish your last adding session? (Y/n): "
  init_response = gets.chomp
end

if (File.size(DBFILE) == 0 || (init_response == "n" || init_response == "N"))
  puts "\nInitializing the database..."
  ActiveRecord::Schema.define do
    begin
      drop_table :friends
    rescue
      # Who cares if we can't drop it...
    end
    create_table :friends do |table|
      table.column :link, :string
      table.column :username, :string
    end
  end
  puts "Done...\n\n"
  adder.do_google_search()
  adder.add_friends()
else
  adder.add_friends()
end

puts "No more friends to add...start a new search."

HTML / CSS to PDF using Ruby on Rails

10 comments Posted by seth Tue, 29 May 2007 21:37:00 GMT

Ever tried to save a web page and send it to someone? What about printing a web page? Both pretty much suck.

A lot of people are talking about printing with HTML and CSS these days but what they don’t tell you is the following:

With CSS you can’t…
  • Determine where page breaks happen.
  • Set page size or type (landscape or portrait).
  • Print background colors…some browsers don’t even print images.
  • Set page footers.

Wow, that’s lame.

Ever tried sending someone a web page? Not a link, but the actual web page. It doesn’t work well at all.

A little history

I wanted to be able to print or send estimates and invoices from our Ruby on Rails application, Cashboard. A good majority of our users felt the same way.

Just sending a link to someone to view a HTML document online doesn’t cut it. What if you want to snail mail that document? Shouldn’t it look the same on the page as it does on your screen?

When I used to create invoices manually I’d always save PDF files for my clients and email them.

PDF handled printing well, was portable, and my clients couldn’t edit the file.

We needed PDF for Cashboard.

The Prince of PDFs

I spent a good week just researching all of the PDF libraries and programs out there.

The ones I found for Ruby wanted me to re-create my documents in some PDF-specific layout language. That wasn’t going to cut it. See, in Cashboard we already had these documents designed in HTML. On top of that, we allow users to customize the colors and upload their logo for invoices. Duplicating the layout wasn’t an option.

After exhausting those possibilities I started looking into HTML to PDF programs. I found a few, but none that were any good at converting HTML and CSS to PDF.

This all changed when I ran across Prince XML. Not only is Prince great at converting HTML and CSS to PDF, it even passed the Acid2 test. They had a ton of demos available on their web site, and provide their program free for evaluation purposes.

Making it happen

Prince is a command line program, available for whatever platform you’re probably running.

It can take a variety of inputs including files on disk, or even HTTP urls. We run a pretty tight ship on Cashboard, and everything is password protected.

Luckilly Prince also can take input from standard in, and can even pass its output back to standard out. This means if you don’t want to mess with saving files and dealing with cleaning them up you don’t have to.

prince.rb, pdf_helper.rb

I cooked up a very simple Ruby library to call Prince and a helper module to include on my Rails controllers. I store them both in the lib folder of my Rails application.

Here’s the full helper, slightly modified for simplicity:

# We use this chunk of controller code all over to generate PDF files.
#
# To stay DRY we placed it here instead of repeating it all over the place.
#
module PdfHelper
  require 'prince'

  private
    # Makes a pdf, returns it as data...
    def make_pdf(template_path, pdf_name, landscape=false)
      prince = Prince.new()
      # Sets style sheets on PDF renderer.
      prince.add_style_sheets(
        "#{RAILS_ROOT}/public/stylesheets/application.css",
        "#{RAILS_ROOT}/public/stylesheets/print.css",
        "#{RAILS_ROOT}/public/stylesheets/prince.css"
      )
      prince.add_style_sheets("#{RAILS_ROOT}/public/stylesheets/prince_landscape.css") if landscape
      # Render the estimate to a big html string.
      # Set RAILS_ASSET_ID to blank string or rails appends some time after
      # to prevent file caching, fucking up local - disk requests.
      ENV["RAILS_ASSET_ID"] = ''
      html_string = render_to_string(:template => template_path, :layout => 'document')
      # Make all paths relative, on disk paths...
      html_string.gsub!("src=\"", "src=\"#{RAILS_ROOT}/public")
      # Send the generated PDF file from our html string.
      return prince.pdf_from_string(html_string)
    end

    # Makes and sends a pdf to the browser
    #
    def make_and_send_pdf(template_path, pdf_name, landscape=false)
      send_data(
        make_pdf(template_path, pdf_name, landscape),
        :filename => pdf_name,
        :type => 'application/pdf'
      ) 
    end
end

This simple module has two methods. Both take an ERB template path as an argument, then a pdf file name.

Make_pdf renders the template to a string, then does some modifications to make all requests within local. I’m passing in all CSS files locally as well, so I don’t have to deal with authentication.

I’ve created some special CSS files for printing, and even can pass in if I’d like the page to be laid out in a portrait or landscape format.

When it’s done it returns the PDF file as data. Nothing is rendered to disk. This method is useful for not only sending the PDF file to the client (as in make_and_send_pdf), but when generating PDF files for email attachments.

Creating PDF files from the controller

Both of these files make creating a PDF file dead easy.

Here’s a slimmed down version of Cashboard’s estimate controller.

class Provider::EstimatesController < Provider::BaseController
  include PdfHelper
    # Sends pdf of an estimate out...
    #
    def pdf
      # @estimate is set with a before_filter and isn't relevant for this how-to ;)
      make_and_send_pdf('/client/estimates/show', @estimate.pdf_name)
    end
end

Does it get any easier than that? Check out an example of a PDF file generated with Prince, straight from Cashboard. (Right click and save to your disk…)

Download

Here again are the libraries I’ve created for using Prince XML with Ruby on Rails:

They’re free to use, download em and check em out.