OpsCode’s VirtualBox VM seems pretty light, sudo du -sh / gives
1.1G. Can I use that image to install stuff on Linode? Or am I better
off starting with Linode’s Ubuntu 12.04 LTS image? I think I probably
want to start with Linode’s image. While looking for other stuff, I
ran across the knife-linode
plugin. Looking at the docs on the Opscode site,
there seem to be configuration parameters for knife-linode that will
let you create a server using one of Linode’s stock images so let’s
see if I can use that to create a box.
To play with it, I needed to go to the Linode “My Profile”
page and create an API key. Then I can list my current servers:
And delete one:
OK so now can I create one?
The user doesn’t seem to have been created. So I can’t ssh in as cnk
(and cnk isn’t in /etc/passwd), but I can ssh in as root. And I don’t
seem to have any ruby or chef - perhaps because I didn’t choose a
distro or template file. Let’s try that again - with --distro
chef-full added.
That give similar output (on a new IP). But when I logged in, I still
don’t have chef or ruby. In fact, there isn’t anything in /opt at
all. Let’s see if I can bootstrap chef by hand.
That seems a bit better; now we have ruby 1.9.3-p484 in
/opt/chef/embedded/bin/ruby. And it came with rubygems 1.8.24, bundler
(1.1.5), rake (10.1.0, 0.9.2.2), and chef (11.8.2). There are a couple
of gems that have more recent versions but I decided not to try
running ‘gem update’ because the installed versions may have been
chosen specifically to satisfy chef dependencies.
I’ll probably want to have chef doing my updates but for now, I think
I should have the server patched so ran apt-get update and apt-get
upgrade by hand for now. (Before I ran the upgrade, I took a list of
the packages that were installed originally with dpkg-query -W >
/root/original-packages.txt so I would know what the Linode Precise
Penguin image came with.)
Connecting my Linode node to Chef server
After installing chef with the knife bootstrap command, I see it tries
to converge the new node. However, there is nothing in my run list -
and no cookbook repository for this new server. Time to make one - and
then put my Linode API configuration into the knife.rb file so I can
quit passing it on the command line all the time.
UGH actually knife seems to have created a node on the server for
me. Was that during knife linode server create? or during the knife
bootstrap command? I can’t really tell from the docs. I should have
passed -N <name> to knife linode server create so that my node
would not end up named ‘localhost’ (and then when that didn’t do the
bootstrapping, again send the name). From what I can tell by Googling,
you can’t really rename a node, though there are delete and readd
proceedures that end up being quite similar. I think I’ll try this one
to get the chef node for my Linode sever named something more sensible
than localhost. I skipped step 6 since I didn’t see ‘localhost’ in
the /etc/chef/client.rb file. Running chef-client -N linode gave:
And now I see a node named “linode” in both the Opscode web interface
and in the output of knife node list. Yeah!
My group at $WORK has decided to start using Django as our web
development framework so I am working my way through
the tutorial.
First off, yeah! there is actually a tutorial! Nice to be using a
mature technology with good documentation.
The overview
does a nice job of showing off some basics - models (including
relationships), routing/urls, views, and templates (including template
composition and reuse). Which brings me to my first mental
mapping. One basic design pattern for web apps is known as MVC - Model
View Controller. So Rails calls three of the directories under ‘app’
models, views, and controllers. There is a fourth directory named
helpers, which is an awkward catch-all that tries to take the logic
out of the view layer, with variable success. And the urls or routes
are not defined within ‘app’ at all and instead are defined in the
config directory in ‘config/routes.rb’. Rails reuses the names from
MVC but in so doing sometimes ends up with some odd compromises. The
Django breakdown, on the other hand, seems to do a nicer job of
splitting the controller and view logic into the logic part (named
views) and the display part, in templates.
One of the best parts of Ruby is rake and Rails makes extensive use of
it for setting up and managing apps. In the Django ecosystem, many
similar tasks are performed via the ‘manager.py’ script.
Command
Rails
Django
Populate the database
rake db:migrate
manage.py syncdb
Shell access to models
rails console
manage.py shell
Global
Django: “reusable apps”
Rails: “engines” (and to a lesser degree, plugins)
Rails: built in sessions but no authentication or authorization.
Django: sessions + authentication. Not sure how much authorization infrastructure comes built in.
Troubleshooting
rails console - great for models. Now better than it was for
controllers, routes, helpers. Not sure about views. Find a recent blog
post and read up / link.
Django shell:
manage.py shell - great for models. To poke around your views layer
(anything that needs the response object), you need to add some power:
Models
Both Rails and Django make handling associations a breeze. In Django,
for one to many associations, all you need to do is create declare an
attribute on the many side (the side that gets the mapping column)
that is a ForeignKey, e.g.:
And Django sets up the mapping for you. Then you can assign a reporter
to an article by saying my_article.reporter = some_reporter. And you
can chain methods to get to the reporter’s name:
my_article.reporter.full_name. From the reporter side, you can get
a list of all her articles by asking for
murrow.article_set.all(). Rails is similar except that you don’t use
foreign key relationships (not my favorite Rails decision) but instead
you declare the relationships explicitly in both models:
The Django database query interface is a bit like the new ActiveModel
syntax - but with ‘get’ instead of ‘find’.
Even in the first part of the tutorial, I run up against Python’s
‘explicit is better than implicit’ philosophy. Rails will use the
class name to infer what instances of your model should be called. You
can get to that value using the human_attribute_name method and you
can either override that method in your model (old school) or use
Rails internationalization capabilities to override the value in your
locale config files. ActiveRecord also provides a default display of
each instance which includes all the attributes of an instance. By
contrast, in Django you need to define a __unicode__() (or in Python 3
a __str__()) method for each model class so that the shell and the
admin interface know what to call things. There doesn’t appear to be a
default representation that shows all the data values; I suspect you
are expected to use Python’s dir() function, e.g. print dir(my_poll).
Models
In Rails the many side of the relationship is accessed as the plural
of it’s name: my_poll.choices. In Django you use type + _set: my_poll.choice_set
Rails scopes (formerly known as named_scopes) let’s you make something
that looks like a method that can be used to constructed queries -
especially useful if you want to chain several of these conditions
together but in a readable way.
Poking around in Django it appears to me that there isn’t such a thing
unless the people answering questions on StackOverflow are as
ignorant as I am about Django’s query interface. The answer appears to
be, if it’s simple, just put the logic for the method directly into
your filter. If it is hairier, possibly write the SQL yourself. ???
Validations? Cool part of Rails. Is there a Django equivalent? or do
you do that validation in the view (aka controller)?
Django admin
Wow! Just wow! Rails scaffold gives you OK CRUD functionality and
there are some plugins that help make things somewhat fancier. But the
out of the box admin functionality in Django is fabulous: sorting,
searching, filtering. The closest thing I know in the Rails ecosystem
is ActiveAdmin, which is nice, but deviates somewhat from the Rails
standard.
One thing I really like a lot is the “History” tab. I end up wanting
to add auditing to a lot of things I write but in several cases I have
added acts_as_versioned to tracked the data - but didn’t provide an
admin interface for displaying the changes. So, of course, now I want
more. Does the Django admin interface provide a way to revert to a
specific version? And can it diff within a text area?
Controllers / Views
Similar render and redirect options. Django provides two really handy
shortcuts get_object_or_404
and [get_list_or_404]((https://docs.djangoproject.com/en/1.8/topics/http/shortcuts/#get-list-or-404).
Handy enough I should create equivalents instead of the controller
code I have.
Routing
Rails’ own interpretation of RESTful urls is strongly favored - in
code and by the community. URLs all defined in a single routes.rb
config file - unless you are using an engine. (For a while engines
didn’t really have a mechanism for creating their routes other than by
having a generator write something into the routes file at
install. But now I think the routing infrastructure will look for
routes files in engines that are mounted into your Rails project.
Django’s routing has a master file, urls.py but it is expected that
each of your site’s apps will define their own urls and that you will
just include those urls into your top level urls.py with a line like
url(r'^polls/', include('polls.urls')) - with or without a
namespace. The point of a namespace is to avoid name conflicts in the
global set of url names. In with the include above, the names in
polls.urls will be global - so the url named details in polls.urls
will be the details url for the entire app. If you say
url(r'^polls/', include('polls.urls', namespace='polls')), then they
will not be global and instead can be accessed as polls:details.
Both frameworks encourage you to use the url lookup mechanism for
creating links instead of hard coding them. The big win for this is
that if your url need to change, you only have to change the
appropriate routes.rb or urls.py file. The references to the urls can
all stay the same.
In both systems, some people like having even shorter syntax for
defining redirects - redirecting to the object. In Rails it is, as
usual, ‘convention over configuration’. If you have defined your
routes using ‘resources’, then you can just say redirect_to object
In Django, you can get a very similar redirect syntax: return
redirect(object) But to make it work, you need to define a
get_absolute_url method on your model class. For example:
Templates
Every web framework provides some way of creating web pages from
modular pieces. In fact defining your design and navigation elements
in one place is sometimes the major reason for using a framework in
the first place. I still need to read ‘The Rails View’ to see if I am
doing this right, but I think the Django views may be a bit more
flexible than the normal Rails view processing.
Rails - one big layout file - chose which main layout file by setting
the name in the controller. ‘yield some_name’ and ‘content_for
some_name’. Helpers - methods to munge display variables.
Django general “extends” mechanism. Named blocks. Filters attached to
data using unix-style pipe, “|”.
Helpers
One of the fabulous things about Rails is all the useful helper
methods available to you. Some were created for internal use but are
super handy for display purposes, e.g. humanize, pluralize,
etc. Others are specifically for display. Many other frameworks either
had their own versions, or borrowed liberally from Rails.
Code generation is nice - but there is a lot less typing involved in
declaring the form using the Django admin tool. Field sets are quite
nice - esp with the automatic ‘collapse’ option. I think this is
fairly similar to Rails’ ActiveAdmin. (Sad to see that AA uses Devise
for it’s authenitcation tool.)
Testing
Both frameworks come from a background of TDD and have pretty good
tools for writing tests. Ruby actually has an embaarssment of riches
since it seems that everyone has written their own testing DSL.
Default task runs all tests, or you can run parts. In Rails there are
tests for each layer. For
Django,
the tests are broken out per app: manage.py test polls. View tests
can assert against the content of the page with ‘contains’ and if you
don’t find what you are looking for, you can print out the web page
content:
Granular testing. Easy to do in rspec_rails. Possible but not so easy
in test_unit/minitest. Easy in Django - just append the name of a
specifit test to your command line:
Database for running tests
Djano: Aside from using a separate database, the test runner will
otherwise use all of the same database settings you have in your
settings file: ENGINE, USER, HOST, etc. The test database is created
by the user specified by USER, so you’ll need to make sure that the
given user account has sufficient privileges to create a new database
on the system.
Both offer fixtures for setting up data. Rails community mainly moved
to factories and the Django tutorial has you create a tiny factory as
a helper methods. Are there libraries for factories in wide use?
Overriding settings - esp urls for apps that could be mounted
anywhere; common option - use a decorator on the test method on or the
entire class: @override_settings(LOGIN_URL=’/other/login/’)
Some useful looking Django test methods:
SimpleTestCase.assertFieldOutput
TestCase.assertFormError
Django integration with Selenium: LiveServerTestCase
I have been using RVM for several years - straight in the terminal and
in a bash shell in emacs. But all of a sudden I started having path
problems. I don’t remember having updated my RVM version, though I
quite possibly I did before I updated my installed rubies for the
latest security patches. Or perhaps there is some odd interaction with
the ruby that got installed when I installed the Heroku toolbelt (v
3.2.0) this last week; it appears to have installed it’s own ruby in
/usr/local/heroku/ruby/bin/ruby but then added /usr/local/heroku/bin
with export PATH="/usr/local/heroku/bin:$PATH" in my .bashrc.
The symptom was that in my bash shell within emacs I was getting
warnings about RVM not finding what it expected at the start of my
path. I should have copied the error message but it was expecting
something like /Users/cnk/.rvm/gems/ruby-2.0.0-p353/bin. Initially I
was getting a path starting with /usr/local/heroku/bin because the
Heroku Toolbelt install had placed that line at the end of my .bashrc
file. But even after having moved the heroku line into my .profile
before the rvm line, I was still getting oddities when trying to use
knive (installed with the chef gem). I tried the fixed suggested in
the warning message: rvm get stable --auto-dotfiles. That gave me
version 1.25.14. and made modifications to my dotfiles:
That made everything work in the terminal but made it much worse in
emacs. With the default rvm setup (source "$HOME/.rvm/scripts/rvm" in
my .bash_profile file), rvm was not getting loaded in my emacs
shell. type -t rvm was returning ‘file’ instead of ‘function’ and
none of my rvm commands would work - no rvm use or rvm gemset
list. I fooled around for a bit and sorted out that what I need to
get rvm to work in the terminal AND in my emacs shell is to have this
line in my .bashrc file NOT in .bash_profile or .profile:
That works fine and does not seem to have any problems even if I
create a new shell within an old one.
Other RVM and Emacs Options
There are a couple of more sophisticated ways of using rvm within
emacs that I came across while trying to figure out why this had
suddenly stopped working. One of these days I should look at them:
I got the same warning again today - when changing directories from
within a shell running in emacs:
This time a) I remembered to record it and b) I read the full message!
I went with the last option: rvm use ruby-2.0.0-p353. Running that
command got me the same warning again - but once the command executed,
then all was repaired and the rest of the commands work fine, e.g. rvm
list, rvm gemset list, etc.
So I have been fooling with chef for nearly a year now - including
going to an Opscode workshop before Rocky Mountain Ruby - but I
haven’t really gotten over the hump into using it for real. But I
think it’s time.
So starting with the Quick Start guide on LearnChef,
I set up my laptop as the workstation. My default ruby under RVM is
currently 2.0.0-p353. I created a gemst called chef and installed the
chef gem (current version is 11.8.2). During the Chef workshop we all
signed up for the free hosted chef server, so I have a login (ckiser)
and organization (ckiser) already. I downloaded the starter kit. (It
warned me that it would reset my user and organization keys but that’s
probably fine.) I moved the chef-repo directory into my home
directory. The README said that stuff there should be under version
control, so I did git init and committed all the files before
starting.
Chef Quick Start - with Vagrant and Chef Server
First off, we installed 2 cookbooks from the Opscode community site,
apt and apache2, with knife cookbook site install [repo]. I am a
little unclear on what ‘site’ means in that command but anyway, that
command created 2 new directories in my cookbooks directory. Then we
created our own cookbook. First we installed a dependency,
magic_shell. Then we generated the cookbook: knife cookbook create
aliases, added the dependency line in the metadata.rb file, and then
edited the default recipe. The tutorial then told us to upload the
cookbooks to our hosted chef using knife cookbook upload --all. It
uploaded all the cookbooks - the ones we got from the community site
and the aliases cookbook we just made - and all without our having to
commit them to git.
But I do have one mystery. The cookbooks we installed using knife
cookbook site install [some repo] do not show up as untracked files
for git - but the aliases cookbook we just created ourselves does. I
don’t see anything obvious like mentions of the community cookbooks in
a .gitignore or .chefignore file. So how does git know the difference?
Then the guide had us boot a VM from the Vagrant file that came with
the starter kit - using an Ubuntu image we downloaded from
OpsCode. The file name, opscode_ubuntu-12.04-i386_provisionerless.box,
makes me think that it doesn’t have chef installed, nor any other
ruby. When I did the initialization that the guide suggested, I see
output that appears to first install chef from OpsCode’s s3 stash:
Hmmm so perhaps I don’t have to preinstall ruby and chef. The
bootstrap process can install the latest version of chef (11.8.2)
packaged with an up to date ruby, ruby 1.9.3p484 (2013-11-22 revision
43786) in /opt/chef/embedded/bin/ruby.
Also note that the bootstrap command, after installing chef,
registered the node we are operating on with my hosted chef
server. The name, ckiser-starter, appears to come from the Vagrant
file that I downloaded at the start of the tutorial,
config.vm.hostname = "ckiser-starter". Are nodes machines? or types
of machines? I think the later, but not sure. I would like to be able
to test out my configuration changes on a local VM before pushing them
to my Linode. So is that 1 node and 2 environments? or 2 nodes? I
think the former but am not 100% sure.
I have been sharing hosting with some old work collegues for a long
while but I am getting the urge to play with some Devops tools. I
don’t want to disrupt them so I need another sandbox. I did most of
the system admininstration for our Rails Rumble team the last couple
of years and thought the Linode VPSs were reasonably speedy and their
configuration web site not too crazy making. And I suspect for the
things I have been doing, their lowest end package will be just fine.
First decision: what Linux distribution do I want to run? I am more
familiar with RPM-based boxes so am tempted to go with either their
Fedora 19 or CentOS 6.4 options. However, the docs for both of those
consider 1G RAM to be a bare minimum for any install. While Ubuntu
quotes 0.5G as their minimum. I don’t know if the difference is a
difference in expectations (is slow acceptable?) but I rather suspect
that Ubuntu is a bit trimmer. So time to suck it up and learn to like
the Debian package management stuff.
This is a server that I want to be pretty stable, so I think the
Precise Penguin 12.04 LTS distribution is what I want. I tried doing a
couple of trial installs from StackScripts and one just from the
configuration manager and all of them installed and then refused to
boot for “unknown reasons”. Not good. I tried booting one in rescue
mode and logging in via the lish console. But I am not really sure if
there is any log file that would show me any information. I don’t
recall exactly where to look but I didn’t see anything while poking
around in things in /var/log/*. One theory is that my problem was
choosing the 32bit distro. Once I capitulated and asked for a 64bit
system, I could build and boot in a variety of ways, just from the
distro, or from either of the following StackScripts:
Minimal Chef Solo
which upgrades all the currently installed packages, then adds the
Opscode apt repository and installs chef from there.
Ubuntu 12.04 - Ruby 1.9.3
which installs the packages needed to compile ruby from source, and
then does just that (Ruby 1.9.3-p194). Or at least it says it does
that. I saw evidence that it had done the downloading and untarring
but ‘which ruby’ gave nothing. Following the steps listed by hand
worked just fine so I am not sure if there was a timing issue or
perhaps something different about running the commands
interactively or what. Anyway mostly works.
OK so what strategy do I want to use? The first one seems generally
easier but installs an older ruby (1.8.7-p352) and older chef
(10.18.2) into /usr/bin. The second one gives me a recent ruby (and I
can make my own script just like it for the latest ruby 1.9.3
version). It also installs into /usr/bin so it will be the global
system ruby. But it is probably good enough to be the ruby I use for
my Rails hosting - without bothering with RVM. Hmmmm let’s look at
setting up Chef before making my final decision.