Vim sessions

Well, I’ve been playing with a nice vim feature: sessions.

As with many others editors out there, vim has support for sessions. How can I save a session with my open files?

:mksession ~/.vim/sessions/project_session.vim

Then you can restore it by:

  1. Starting vim with the -S param:

    vim -S ~/.vim/sessions/project_session.vim

  2. In command mode:

    :source ~/.vim/sessions/project_session.vim

To facilitate further, you can put some functions and mappings in your config file.

I assume that you will start in the top directory of your project as the commands are relative to it (it names the session file with this directory name):

if !exists("g:session_dir")
  let g:session_dir = $HOME . "/.vim/sessions"
endif

function! SaveSession()
  let name = split(getcwd(), "/")[-1]
  let session_file = g:session_dir . "/" . name . '.vim'
  execute 'mksession! ' . session_file
  echom "Session ". name . " saved!"
endfunction

function! RestoreSession()
  let name = split(getcwd(), "/")[-1]
  let session_file = g:session_dir . "/" . name . '.vim'
  if filereadable(session_file)
    execute 'so ' . session_file
      if bufexists(1)
        for l in range(1, bufnr('$'))
          if bufwinnr(l) == -1
            exec 'sbuffer ' . l
          endif
        endfor
      endif
   endif
   echom "Session " . name . " loaded!"
endfunction

nmap <leader>ss :call SaveSession()<CR><CR>
nmap <leader>sr :call RestoreSession()<CR><CR>

That way, you just have to :cd <dir-project> and <leader>sr.

Well, that’s it. It can be improved further (a lot) but these are some basics. And, of course, there are some plugins to deal with sessions too: vim-session, autosess, and maybe others…

I’ve compiled things you people wouldn’t believe. Garbage collectors on fire off the shoulder of Orion. I watched optimizations glitter in the dark near the Tannhäuser Gate. All those moments will be lost in time, like a broken nightly. Time to deploy.
Anybody knows who is the author?

Riders on the Storm

Yesterday I was checking that amazing piece of software: Storm.

Though it was a long time since my last Java work it was easy to get in touch with Storm.

In this post, I just want to explain the steps I’d take to install and run the first tests, just for convenience (almost for me).

The tools

  1. CentOS 6.4
  2. Java 7 (OpenJDK)
  3. OSSP uuid
  4. Zookeeper (installed from the Cloudera repo)
  5. Storm-installer

Steps

With a clean server, install CentOS 6.4 (I used the minimal installtion), add EPEL repository, add Cloudera repository and upgrade

rpm -ivh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -ivh http://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm
rpm --import http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera
yum upgrade

Install development group and Base

yum groupinstall -y “Development tools” Base

Install JDK

yum install -y java-1.7.0-openjdk-devel

Install uuid

yum install -y uuid

Install Zookeeper

yum install -y zookeeper-server

For zookeeper server to start, there must be an id defined:

echo 1 > /var/lib/zookeeper/myid

Start zookeeper

service zookeeper-server start

(there is a service zookeeper-server init that should be executed previously but in my case it complains that the /var/lib/zookeeper directory exists)

You can check if it is up and running connecting with the provided tool:

/usr/lib/zookeeper/bin/zkCli.sh -server localhost:2181

Download and install storm with storm-installer

wget https://www.dropbox.com/s/ggcmy9hjyvq3abd/storm-installer-0.8.2_1.el6.x86_64.zip -O storm-installer-0.8.2_1.el6.x86_64.zip

Follow the instructions and install storm plus dependencies

unzip storm-installer-0.8.2_1.el6.x86_64.zip
cd storm-installer-0.8.2_1.el6.x86_64
rpm -ivh zeromq-2.1.7-1.el6.x86_64.rpm
rpm -ivh zeromq-devel-2.1.7-1.el6.x86_64.rpm
rpm -ivh jzmq-2.1.0-1.el6.x86_64.rpm
rpm -ivh jzmq-devel-2.1.0-1.el6.x86_64.rpm
rpm -ivh storm-0.8.2-1.el6.x86_64.rpm
rpm -ivh storm-service-0.8.2-1.el6.x86_64.rpm
updatedb

Make the init scripts available to service command

vim /etc/init.d/storm-nimbus
[...]
### BEGIN INIT INFO
# Provides: $storm-nimbus
# Required-Start: $network $local_fs $zookeeper-server
# Default-Start: 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Nimbus Storm Manager
### END INIT INFO

vim /etc/init.d/storm-supervisor
### BEGIN INIT INFO
# Provides: $storm-supervisor
# Required-Start: $storm-nimbus
# Default-Start: 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Storm Workers Supervisor Manager
### END INIT INFO

Note that Required-Start require $storm-nimbus. This is because I am running the test in a single machine but can not be true in a cluster where you will run the supervisor in a machine withouth nimbus.

vim /etc/init.d/storm-ui
### BEGIN INIT INFO
# Provides: $storm-ui
# Required-Start: $storm-nimbus
# Default-Start: 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Storm Web Interface
### END INIT INFO

Check the storm init file

vim /opt/storm/conf/storm.yaml

Be sure the configuration match your needs.

Start storm and verify

service storm-nimbus start
service storm-supervisor start
service storm-ui start

All should be working now. Check it:

ps ux | grep storm

Add to init

If everything is ok, add the services to start at boot

chkconfig storm-nimbus on
chkconfig storm-supervisor on
chkconfig storm-ui on

Adjust firewall to allow the thrift port (upload topologies)

iptables -I INPUT 1 -m state --state NEW -m tcp -p tcp --dport 6627 -j ACCEPT
iptables-save > /etc/sysconfig/iptables

Everything should be working now. Lets try it wiht storm-starter:

Install lein script

cd /usr/local/bin
curl -O https://raw.github.com/technomancy/leiningen/stable/bin/lein
chmod +x lein

As I am running this as root (humm, don’t do it), we have to:

echo "export LEIN_ROOT=t" >> $HOME/.bashrc

Go to your projects dir and clone storm-starter

git clone git://github.com/nathanmarz/storm-starter.git
cd storm-starter

Get deps and compile

lein deps
lein compile

Try it

lein repl
  (import `storm.starter.ExclamationTopology)
  (ExclamationTopology/main nil)

We use the REPL method to run the topologies but you could also build the har and excecute from command line:

lein jar
java -cp `lein classpath` storm.starter.ExclamationTopology

You also can try the other examples included: WordCountTopology and ReachTopology.

Final thougths

All went smooth and working fine. But there is a couple things I want to correct in a near future.

First, if zookeeper or nimbus stop working we have a problem. I want to add some security by starting the services via upstart, supervisord or monitoring with god.

Then I will install from sources letting the installer as an option. The good about the installer is that you do not need the development tools on the server, and this is good.

Finally I would like to make some Chef recipes to install the whole thing.

Enough by now. Time to deploy.

Leaving Facebook (forever)

I’ve just deleted my facebook account. I have no friends anymore ;)

Why? I have the impression of losing control about what is published and who can access. I don’t have the time to invest in knowing and securing it, so I just leave.

Btw, I followed this guide. Thanks.

Some vim niceties

Well, these are some picks I use often in vim. Just to remember them…

I will be updating this post wherever I find something worth remembering.

How to open multiple files in tabs

:args /my/path/*.erl
:tab all

Now you have loaded the files referenced in the var args in his very own tab.

Change directory to the one of the file we are editing

There are plenty of methods in vim tips wiki. The one I use is

nnoremap ,cd :cd %:p:h<CR>:pwd<CR>

in my .vimrc file. This way its easy to ,cd whenever I need it.

BTW, you can go back with

:cd -

Reload a file using a different encoding

Sometimes I want to change file encoding. When I have a file in utf-8 and coding in Erlang I want it in latin1 just in case. This can be easily done:

:e ++enc=<encoding>

You can map a key if you use it often

" Press F12 to switch to UTF-8 encoding
nnoremap <F12> :e ++enc=utf-8<CR>

Also, you can change the encoding as

:set fileencoding=<encoding>

to change the encoding that will be used to save the file. To change the encoding shown in the terminal:

:set encoding=<encoding>

Finally, you can also change as you write:

:write ++enc=<encoding>

To find valid encoding values, read vim documentation.

Run unattached background jobs in Linux

If you are logged in your Linux shell account, there are some things you can do.

The easyest &

Every command you run ending with an ampersand will be backgrounded by the shell

$ my_command &

BUT, if you ever logout while it is running, it will die because the shell will send a SIGHUP to all depending jobs, including yours.

Fortunately, there are solutions for that.

nohup

Running the command through nohup it will survive your logouts. If your command is not already running, launch it with nohup.

$ nohup my_command > output.log 2>&1&

This will prevent that the shell send it SIGHUP on logout (or lose connection) so your command will keep running.

disown (if you’re using bash)

If you just started running your command and still want to protect it from being signaled, bash shell provides a builtin that fits the job: disown.

With disown you can prevent a running job from being signaled or even remove it from the jobs table. In any case, it will keep running after your shell ends.

$ yes > /dev/null &
[1] 1208
$ ps x | grep yes
  1208 s007  R      0:27.41 yes
$ jobs
[1]+   Running                yes > /dev/null &
$ disown -h %1

Now, you can see that the command is still executing usign another builtin, jobs as shown above.

If you call disown withouth the -h switch, it will remove the entry from the jobs table and you will not see it with jobs (but still show with ps).

Also if you do not provide the job number, the last one will be taken.

How to do it if the process is already running in foreground?

Well, not much trouble. You have to stop it, resume it in background and finally disown it. This are features provided by the bash job control.

$ yes > /dev/null
^Z (Ctrl-Z)
[1]+  Stopped                yes > /dev/null
$ jobs
[1]+  Stopped                yes > /dev/null
$ bg %1
[1]+  yes > /dev/null &
$ jobs
[1]+  Running                yes > /dev/null &
$ kill -9 %1
[1]+  Killed: 9                yes > /dev/null

Other alternatives

You also can execute a terminal multiplexer that takes care of that. I use regularly tmux but screen is another popular one.

When you start tmux | screen, you can start working in multiple windows. If the connection is lost, when you reattach jus call them indicating that you want to reattach to the active session:

$ tmux attach

or

$ screen -D

That’s it. Never get abruptly interrupted again!

FC16. Build native pg gem

While trying to install the gems required by my project I got an error while installing pg:

ERROR: Failed to build gem native extension.

First of all, I made sure that the postgresql development package was installed (it was). I’m using the RPMs provided by pgdg so I just

rpm -aq | grep postgresql

to confirm. If you don’t have it

yum install postgresql91-devel

So the error was about the install script not founding pg_config which its installed under

/usr/pgsql-9.1/bin

Just install with

env PATH=$PATH:/usr/pgsql-9.1/bin bundle install

and everything went as expected.

Howto burn a bootable USB under OSX

The best and accepted method is the one provided by Ubuntu that I reproduce here. Can be used with any bootable iso image (a live operating system, a utility cd, whatever).

This method provides a bootable USB on any PC. Macs use EFI (instead of BIOS) so this will not work as is. If wanna boot a Mac, you better follow this one.

Burning from the shell

This steps are from the Ubuntu download page (select USB and the Show me how button).

  1. Download the desired file
  2. Open the Terminal (in /Applications/Utilities/ or query Terminal in Spotlight)
  3. Convert the .iso file to .img using the convert option of hdiutil (e.g., hdiutil convert -format UDRW -o ~/path/to/target.img ~/path/to/ubuntu.iso)
  4. Note: OS X tends to put the .dmg ending on the output file automatically.
  5. Run diskutil list to get the current list of devices
  6. Insert your flash media
  7. Run diskutil list again and determine the device node assigned to your flash media (e.g. /dev/disk2)
  8. Run diskutil unmountDisk /dev/diskN (replace N with the disk number from the last command; in the previous example, N would be 2)
  9. Execute sudo dd if=/path/to/downloaded.img of=/dev/rdiskN bs=1m (replace /path/to/downloaded.img with the path where the image file is located; for example, ./ubuntu.img or ./ubuntu.dmg).
    • Using /dev/rdisk instead of /dev/disk may be faster.
    • If you see the error dd: Invalid number '1m', you are using GNU dd. Use the same command but replace bs=1m with bs=1M.
    • If you see the error dd: /dev/diskN: Resource busy, make sure the disk is not in use. Start the ‘Disk Utility.app’ and unmount (don’t eject) the drive.
  11. Run diskutil eject /dev/diskN and remove your flash media when the command completes
  12. Restart your Mac and press alt while the Mac is restarting to choose the USB-Stick

Now you have a bootable USB-Stick that can use in any computer.

I’ve used this method many times and always worked as expected. Nice job.

Utilities

There are other tools that can do the job. You can try:

Anyway, I feel more comfortable at the command line.

Make GVim the default Windows Editor

Today I installed a new environment under Windows 7(R) but I’m not prepared to lose my vim. Install vim under Windows it’s an easy task, just download and install from http://www.vim.org. I installed v7.3 (go get it).

Once installed, I removed notepad.exe and replaced it by gvim.exe. Here we go:

First, turn off User Account Control by running uac from the search box. Drag the slider down to Never Notify.

Now, we have to set an environment variable to tell gvim where the install directory is located. If you don’t do it gvim will not work as expected.

Go to Panel de control\Todos los elementos de Panel de control\Sistema and click on Configuración Avanzada del Sistema. Now on Variables de entorno… and set the variable VIMRUNTIME to 

C:\Program Files (x86)\Vim\vim73

Or wherever you have vim installed.

Time to reboot. Hey! Its windows after all!

Once back, we have to replace the executable notepad.exe with gvim.exe (based on Replace Notepad in Windows 7) by running

takeown /f c:\windows\syswow64\notepad.exe
cacls c:\windows\syswow64\notepad.exe /G Administradores:F
takeown /f c:\windows\system32\notepad.exe
cacls c:\windows\system32\notepad.exe /G Administradores:F
takeown /f c:\windows\notepad.exe
cacls c:\windows\notepad.exe /G Administradores:F

copy c:\windows\syswow64\notepad.exe c:\windows\syswow64\notepad.exe.backup
copy c:\windows\system32\notepad.exe c:\windows\system32\notepad.exe.backup
copy c:\windows\notepad.exe c:\windows\notepad.exe.backup

copy "C:\Program Files (x86)\Vim\vim73\gvim.exe" c:\windows\syswow64\notepad.exe
copy "C:\Program Files (x86)\Vim\vim73\gvim.exe" c:\windows\system32\notepad.exe
copy "C:\Program Files (x86)\Vim\vim73\gvim.exe" c:\windows\notepad.exe

My windows installation is in Spanish so do not copy&paste.

And thats it. Now you’re a happy vim user getting rid of Notepad.

Congrats!

Migrate repository from Subversion to Git

Its time to move previous subversion repositories to git.

I thougth that it would be more painfull than it was thanks to John Albin great post about the subject. You can read it and follow here: http://john.albin.net/git/convert-subversion-to-git

I just had to add the —follow-parent option in step two or I ended with an empty dir.

Similar thing in step three, do not work withouth -i trunk.

After Step 3 I stopped with the Albin way. I am using Gitorious to deal with Git repositories so I created a new project, a new repository and then commited to it.

And everything is working nice.

Here follows a little summary of the steps I take:

Step 1. Retrieve a list of all Subversion committers

$ svn log -q | awk -F '|' '/^r/ {sub("^ ", "", $2); sub(" $", "", $2); print $2" = "$2" <"$2">"}'| sort -u > /tmp/authors-transform.txt

Step 2. Clone the subversion repository.

$ git svn clone http://svn.server/backup_rsync --no-metadata -A /tmp/authors-transform.txt --follow-parent --stdlayout /Users/local_user/tmp/migrate_from_svn

Step 3. Convert svn:ignore properties to .gitignore

 
$ cd /Users/local_user/tmp/migrate_from_svn 
$ git svn show-ignore -i trunk > .gitignore 
$ git add .gitignore 
$ git commit -m 'Convert svn:ignore properties to .gitignore.'

Step 4. Create a project and a repository at gitorious.

Step 5. Push the local repository to gitorious.

 
$ git remote add origin git@gitorious.server:zfs-backup/backup_rsync.git 
$ git push origin master 
Counting objects: 826, done. 
Delta compression using up to 4 threads. 
Compressing objects: 100% (748/748), done. 
Writing objects: 100% (826/826), 117.15 KiB, done. 
Total 826 (delta 484), reused 0 (delta 0) 
 remote: => Syncing Gitorious... [OK] 
To git@gitorious.server:zfs-backup/backup_rsync.git 
 * [new branch]      master -> master

Slow SSH login

Sometimes when I try to login to a server via ssh it takes a loooooong time to ask me for the password to log in. After quite some time, I decided to fix it because it is getting really annoying.

It seems that the problem is related to GSSAPI authentication that takes a long time trying to do a reverse DNS resolve for the calling host.

Solution(s):

  • Prevent sshd to look into DNS. You cant edit sshd_config file and set UseDNS no. You can also add the host in /etc/hosts file.
  • Use the command line option -o GSSAPIAuthentication=no in the client:
    $ ssh -o GSSAPIAuthentication=no my_user@my.remote.host
  • Disable GSSAPI in the server. Edit /etc/ssh/sshd_config (or wherever it is the file) and set GSSAPIAuthentication no (check that the option is not enable somewhere else in the file). Restart sshd.
  • Disable GSSAPI in the client. Either set GSSAPIAuthentication no in /etc/ssh/ssh_config or create an $HOME/.ssh/config file with the option set.

Usually I disable GSSAPI in the server and forget about it but the other options comes to rescue when you do not own the server.