(blog ‘lucindo)

marvin:~ lucindo$ python
Python 2.5.2 (r252:60911, Apr 26 2008, 14:32:15)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
>>>

Bom, esse post é mais um snippet de como implementar um serviço REST usando Hunchentoot. Dessa vez adicionei suporte a ETags no código do outro post. Ficou assim o tratamendo das requisições:

(defvar *data-id* “e8d8993494ffc11:b8e”)
(defun get-data-id (target)
  “get the last id for requested data”
  *data-id*)

(defmethod handle :around (request-method)
  “ETag support for all methods”
  (let ((data-id (get-data-id (script-name)))
        (last-id (header-in :If-None-Match)))
    (setf (header-out :ETag) data-id)
    (if (and last-id (equalp last-id data-id))
        (setf (return-code) +http-not-modified+)
        (call-next-method))))

Fazendo um request normal:

GET /rest HTTP/1.1
Host: eddie

HTTP/1.1 200 OK
Content-Length: 3
Content-Type: text/html; charset=iso-8859-1
Date: Sun, 27 Apr 2008 20:07:53 GMT
Server: Hunchentoot 0.15.6
Etag: e8d8993494ffc11:b8e

GET

Agora o cliente passando o ETag:

GET /rest HTTP/1.1
Host: eddie
If-None-Match: e8d8993494ffc11:b8e

HTTP/1.1 304 Not Modified

Suportar ETags é muito fácil e se o cliente do seu serviço souber usar é muito útil para os dois.

Download: rest.lisp

As to your real question, the idea of immediate compilation and “unit tests” appeals to me only rarely, when I’m feeling my way in a totally unknown environment and need feedback about what works and what doesn’t. Otherwise, lots of time is wasted on activities that I simply never need to perform or even think about. Nothing needs to be “mocked up.”

Donald Knuth em entrevista a InformIT

Como programador ele é um ótimo businessman

Um snippet de teste que eu fiz para um serviço REST usando Hunchentoot.

Usando função genérica é possível fazer um “pattern matching” e a implementação fica parecida com Erlang/YAWS (a vir no BedDB).

(eval-when (:load-toplevel :compile-toplevel :execute)
  (require ‘hunchentoot))

(defpackage :ht-rest
  (:use :common-lisp :hunchentoot))

(in-package :ht-rest)

(defun add-dispatcher (dispatcher-fn)
  “Helper function to add dispatcher functions to dispatch table”
  (nconc *dispatch-table* (list dispatcher-fn)))

(defun create-dispatcher (url-prefix handler &key (regexp nil))
  “Creates a dispatcher and add it to dispatch table given the
url prefix and handler function. The url prefix can be a regular
expression (in this case set the :regexp keyword).”
  (let ((dispatcher-fn
         (funcall
          (if regexp ‘create-regexex-dispatcher ‘create-prefix-dispatcher)
          url-prefix handler)))
    (add-dispatcher dispatcher-fn)))

(defun handle-rest ()
  “Simply delegate to appropriate handler”
  (handle (request-method)))

(defgeneric handle (request-method)
  (:documentation “Generic REST handler”))

(defmethod handle :before (request-method)
  (log-message :info “REST in: [method ~a] [target ~a] [qs ~s]”
               (request-method) (script-name) (query-string)))

(defmethod handle :after (request-method)
  (log-message :info “REST out: [code ~a] [method ~a] [target ~a] [qs ~s]”
               (return-code) (request-method)
               (script-name) (query-string)))

(defmethod handle ((request-method (eql :get)))
  (string (request-method)))

(defmethod handle ((request-method (eql :post)))
  (string (request-method)))

(defmethod handle ((request-method (eql :put)))
  (string (request-method)))

(defmethod handle ((request-method (eql :delete)))
  (string (request-method)))

(defmethod handle (request-method)
  (setf (return-code) +http-method-not-allowed+))

(defvar *ht-server* nil)

(defun setup ()
  (create-dispatcher “/rest” ‘handle-rest))

(defun start (&optional (setup-p t))
  (prog1
      (setq  *show-lisp-errors-p* t
             *show-lisp-backtraces-p* t
             *dispatch-table* (list ‘dispatch-easy-handlers)
             *ht-server* (start-server :port 8080))
    (when setup-p (setup))))

(defun stop ()
  (stop-server *ht-server*))

(defun re-start ()
  (progn
    (stop)
    (start nil)))

    Computer science is not glorified programming. Edsger Dijkstra, Turing Award winner and extremely opinionated man, famously said that computer science has as much to do with computers as astronomy has to do with telescopes. We claim that computer science is a mathematical set of tools, or body of ideas, for understanding just about any system - brain, universe, living organism, or, yes, computer. Scott got into computer science as a kid because he wanted to understand video games. It was clear to him that if you could really understand video games then you could understand the entire universe. After all, what is the universe if not a video game with really, really realistic special effects?
    OK, but isn’t physics the accepted academic path to understanding the universe? Well, physicists have what you might call a top-down approach: you look for regularities and try to encapsulate them as general laws, and explain those laws as deeper laws. The Large Hadron Collider is scheduled to start digging a little deeper in less than a year.
    Computer science you can think of as working in the opposite direction. (Maybe we’ll eventually meet the physicists half-way.) We start with the simplest possible systems, and sets of rules that we haven’t necessarily confirmed by experiment, but which we just suppose are true, and then ask what sort of complex systems we can and cannot build.

Scott Aaronson - Great Ideas In Theoretical Computer Science Lecture 1

Passo a passo para instalar e configurar Erlang, SBCL e Emacs no Leopard.

Primeiro instale o xcode que vem no CD do Leopard. Em seguida instale o MacPorts.

O MacPorts está com versões bem atualizadas de Erlang (R12B-0) e SBCL (1.0.13):
$ sudo port selfupdate
$ sudo port install erlang +smp +ssl
$ sudo port install sbcl +threads
$ sudo port clean --all installed


O MacPorts instala tudo no /opt, então coloque o seguinte no seu .profile:
export PATH=/opt/local/bin:/opt/local/sbin:$PATH


Para emacs existem algumas opções. Eu uso o Carbon Emacs (que já vem com slime). Depois de instalar adicione o seguinte no seu .emacs:

(add-to-list 'exec-path “/opt/local/bin”)
(add-to-list ‘load-path “/opt/local/lib/erlang/lib/tools-2.6/emacs”)

(setq erlang-root-dir “/opt/local”)
(require ‘erlang-start)

(require ‘slime)
(setq inferior-lisp-program “sbcl –noinform”)
(add-hook ‘lisp-mode-hook (lambda () (slime-mode t)))
(add-hook ‘inferior-lisp-mode-hook (lambda () (inferior-slime-mode t)))

Update: Para passar parâmetros para a Erlang VM use o seguinte no seu .emacs:

(setq inferior-erlang-machine-options
      '(“-pa” “/opt/local/lib/yaws/ebin”
        “-sname” “mini”))

Para quem ainda não sabe, o Netflix Prize é uma competição que começou no final do 2006 cujo objetivo é melhorar em 10% o algoritmo de sugestão de filmes da Netflix. Quem conseguir fazer isso recebe o prêmio de U$ 1.000.000.

Bom, fiz um pequeno código esses dias que joga os dados do Netflix Prize no Postgres para depois poder brincar um pouco com isso e quem sabe ganhar uma graninha

Usei basicamente o Postmodern e a nova versão do Postmodern-utils. Ainda preciso fazer uma página com a documentação do pacote. Nessa versão entraram algumas funções novas: set-connection-spec, create-tables, map-dao e for-each-dao.

Seguindo a sugestão do Leonardo Varuzza, agora o Postmodern-utils poder ser instalado assim:

* (asdf-install:install :postmodern-utils)

Vamos ao código. Primeiro a definição do pacote:

(eval-when (:load-toplevel :compile-toplevel :execute)
  (require :postmodern)
  (require :postmodern-utils)
  (require :cl-fad)
  (require :split-sequence))

(defpackage :netflix
  (:use :common-lisp :postmodern :postmodern-utils)
  (:export #:setup-database
           #:load-data
           #:predict-ratings
           #:movies #:id-of #:year-of #:title-of #:get-movie
           #:ratings #:user-of #:movie-of #:rating-of #:date-of
           #:get-ratings-of-user #:get-ratings-of-movie))

(in-package :netflix)

Agora alguns dados globais (mude o parametro *netflix-data-dir*):

(defparameter *netflix-data-dir* “/path/to/netflix/download”)
(defparameter *movies-file*
  (concatenate ’string *netflix-data-dir* “/movie_titles.txt”))
(defparameter *ratings-dir*
  (concatenate ’string *netflix-data-dir* “/training_set/”))

(eval-when (:load-toplevel :compile-toplevel :execute)
  (set-connection-spec :username “netflix”
                       :password “”
                       :database “netflix”
                       :hostname “localhost”))

Agora a definição da tabela movies e funções para criar e acessar daos desse tipo:

(deftable movies ()
  ((movie-id :type integer
             :accessor id-of
             :initarg :movie-id)
   (year :type string ;; may be “NULL”
         :accessor year-of
         :initarg :year)
   (title :type string
          :accessor title-of
          :initarg :title))
  (:indices movie-id)
  (:class-name movies))

(defun make-movie (movie-id year title)
  (with-pooled-connection
      (save-dao (make-instance ‘movies
                               :movie-id movie-id
                               :year year
                               :title title))))

(defun get-movie (movie-id)
  (with-pooled-connection
      (get-dao ‘movies movie-id)))

Para ler e parsear os arquivos vamos usar a seguinte função auxiliar:

(defun process-file-by-line (file-name fn)
  (with-open-file (stream file-name :external-format :iso-8859-1)
    (do ((line (read-line stream nil)
               (read-line stream nil)))
        ((null line))
      (funcall fn line))))

Com ela podemos parsear o arquivo movie_titles.txt assim:

(defun load-movies (movies-file)
  (process-file-by-line
   movies-file
   #’(lambda (line)
       (let* ((splitted-line (split-sequence:split-sequence #\, line))
              (movie-id (parse-integer (first splitted-line)))
              (year (second splitted-line))
              (title (third splitted-line)))
         (make-movie movie-id year title)))))

Depois definimos a tabela ratings e uma função para criar os daos desse tipo:

(deftable ratings ()
  ((user-id :type integer
            :accessor user-of
            :initarg :user-id)
   (movie-id :type integer
             :accessor movie-of
             :initarg :movie-id)
   (rating :type integer
           :accessor rating-of
           :initarg :rating)
   (date :type string
         :accessor date-of
         :initarg :date))
  (:indices (user-id movie-id) rating)
  (:class-name ratings))

(defun make-rating (user-id movie-id rating date)
  (with-pooled-connection
      (save-dao (make-instance ‘ratings
                               :user-id user-id
                               :movie-id movie-id
                               :rating rating
                               :date date))))

Podemos fazer algumas funções para selecionar listas de ratings, como por exemplo:

(defun get-ratings-of-user (user-id)
  (with-pooled-connection
    (select-daos ‘ratings :test `(:= user-id ,user-id))))

(defun get-ratings-of-movie (movie-id)
  (with-pooled-connection
    (select-daos ‘ratings :test `(:= movie-id ,movie-id))))

O parse dos arquivos com ratings é feito pelas seguintes funções:

(defun load-ratings-file (file)
  (let ((movie-id 0))
    (process-file-by-line
     file
     #’(lambda (line)
         (if (search “:” line)
             (setq movie-id (parse-integer line :junk-allowed t))
             (let* ((splitted-line (split-sequence:split-sequence #\, line))
                    (user-id (parse-integer (first splitted-line)))
                    (rating (parse-integer (second splitted-line)))
                    (date (third splitted-line)))
               (make-rating user-id movie-id rating date)))))
    (format t “processed: ~a~%” file)))

(defun load-ratings (ratings-dir)
  (cl-fad:walk-directory
   ratings-dir
   #’(lambda (file)
       (load-ratings-file file))))

Agora as funções para carregar todos os dados no Postgres:

(defun load-data ()
  (load-movies *movies-file*)
  (load-ratings *ratings-dir*))

(defun setup-database (&optional (first-drop nil))
  (with-pooled-connection
    (create-tables ‘(movies ratings) :first-drop first-drop)))

Por fim a função para processar os arquivos de entradas e gerar o arquivo com as previsões dada uma função de previsão que recebe três parêmetros: id do usuário, id do filme e a data da votação.

(defun predict-ratings (query-file predict-file predict-fn)
  (with-open-file (stream predict-file :direction :output)
    (let ((movie-id 0))
      (process-file-by-line
       query-file
       #’(lambda (line)
           (if (search “:” line)
               (progn
                 (setq movie-id (parse-integer line :junk-allowed t))
                 (format stream “~a:~%” movie-id))
               (let* ((splitted-line (split-sequence:split-sequence #\, line))
                      (user-id (parse-integer (first splitted-line)))
                      (date (second splitted-line)))
                 (format stream “~1$~%”
                         (funcall predict-fn user-id movie-id date)))))))))

Agora tenho todos os dados e funções para fazer a manipulação. Só falta a função de 1 milhão de dólares e rodar:

(predict-ratings “qualifying.txt” “winner.txt” #’million-dollar-function)

Download: netflix.lisp

Observação 1: carregar todos os dados no Postgres dessa maneira demora (poucos dias), pois faço um insert para cada rating, e são 100 milhões deles.

Observação 2: vale a pena ver também o excelente post do Juho Snellman sobre como processar os dados do Netflix Prize usando Common Lisp.