31 December 2010

Proxying Server from Apache on Synology

How to redirect a part of your web site to another server on your LAN:

... On your NAS: at the bottom of /usr/syno/apache/conf/httpd.conf-user

ProxyRequests Off
<proxy *>
  Order deny,allow
  Allow from all
</proxy>
ProxyPass /redmine http://192.168.0.78/redmine
ProxyPassReverse /redmine http://192.168.0.78/redmine
<location redmine>
  AuthName "Protected Access"
  AuthType Basic
  AuthUserFile /var/services/web/hg/.htpasswd
  require valid-user
</location>



It tells Apache to redirect all /redmine/* access to another server (this subdirectory is protected by password in this case).

And... That's all :)

Virtual machine as Windows service

A follow up from
Synology as Web Gateway (part one)
Once you have a virtual server up and running in Virtualbox, the question of starting it automatically in the background may be solved by the use of ServiceMan.

ServiceMan is a small win32 app i wrote as Virtualbox (3.xx) is not natively service-able. It features execution of a set of cmd lines, and on-the-fly environment variables setting. Both are needed to start a virtualbox machine (plus is able to launch many apps in one go, and support internal variables).

You need to type a cmd line to register ServiceMan as a service (it will be executed at host start).
ServiceMan -i [ServiceName]

ServiceMan will get all information from a old-fashioned ServiceMan.ini located in the executable directory. Here is mine in this case:



[General]
;this shows in services control panel
Description=ServiceMan (a virtual machine using virtualbox)

;
;From now, each section is considered to be triggered sequentially
;at host startup and shutdown
;
;We're launching a virtual server in this one
[VirtualUbuntu]

;
;Aliases : allow to define internal variables you'll be reusing later in config file
;here i'm using a portable edition of virtualbox, and need to set some paths
;
Aliases=VboxUserHome=D:\Machines\Portable-VirtualBox\data\.VirtualBox;VboxDir=D:\Machines\Portable-VirtualBox\app64

;A more conventional virtual box installation should use this
;Aliases=VboxUserHome=C:\Documents and Settings\[User Name]\.VirtualBox;VboxDir=C:\Program Files\Oracle\VirtualBox

;
;Environment variables to add to your configuration when executing cmd lines
;
EnvVars=VBOX_USER_HOME=%VboxUserHome%;VBOX_INSTALL_PATH=%VBoxDir%

;
;Working directory
;
WorkingDir=%VboxDir%

;
;we can redefine envvars for that particular service event
;with StartupEnvVars=VBOX_USER_HOME=%VboxUserHome%;VBOX_INSTALL_PATH=%VBoxDir%
;following are the command executed sequentially. It waits 1000s by default
;
;doing this should force vboxsvc to read new env vars
StartupCmd1="%VboxDir%\vboxsvc.exe" /reregserver
;it's possible to define a pause between commands (default is 1000 ms)
;StartupPause1=1000
StartupCmd2="%VboxDir%\vboxmanage.exe" list systemproperties
StartupCmd3="%VboxDir%\vboxmanage.exe" list vms
StartupCmd4="%VboxDir%\vboxheadless.exe" -startvm VirtualLinux

;
;and shutdown event: again, if we need to redefine things...
;ShutdownEnvVars=VBOX_USER_HOME=%VboxUserHome%;VBOX_INSTALL_PATH=%VBoxDir%
;ShutdownWorkingDir=%VboxDir%
;
ShutdownCmd1="%VboxDir%\vboxmanage.exe" controlvm VirtualLinux savestate

;
;you may have more than one app to launch as a service: open another section
;[AnotherSetOfCommand]
;etc, etc
;



If you need to start and stop for testing, use the service control panel or the cmd line -s / -k option.

Note that it generates a session exec log in ServiceMan directory (which, therefore, should be writable).

Download ServiceMan

24 November 2010

Synology as Web gateway (part1)

RedMine is a great tool for developpers: unfortunately it does not perform well on my synology DS-107+ (for various reasons i don't have time to inspect, low memory is probably a first answer)...

Anyway, as i'm having a-lot-of (probably too-much) machines available @ home: i choose to dedicate some horsepower to run a minimalistic virtual linux machine with Apache and RedMine (note that Bitnami is offering such packages ready for download).

So my idea was to use the Synology as a gateway between the internet and my home LAN, mainly to centralize security settings, and access point uniqueness.

Many topics are covered in this recipe:

1) Installing a virtual ubuntu server running redmine
2) Running a virtual machine as a service under Windows 7
3) Binding the Synology web frontend and a virtual server

Creating the virtual machine : ubuntu+redmine
Get VirtualBox, http://www.virtualbox.org/ . Eventually look for a portable edition.
    Create a new virtual machine: Ubuntu type, i dedicated 20gb of expandable disk space and 512Mo memory (but 5Go and 256Mo should be fine). Use Bridged Network Access for network connections (it should allow the virtual machine to have its own static IP address on your LAN).
      Note: to move a virtual server from one machine to another: ensure that you're using the same netword card between virtualbox installations: type of network card and MAC address must be the same. (note that if network is not working properly on virtualized linux, ensure that /etc/udev/rules.d/70-persistent-net.rules doesn't contain deprecated MAC entries).
      Grab Ubuntu server ISO, attach it to your virtua machine and perform the (blazingly fast) install: select Apache and MySQL components
        • Once connected to the console, you might notice a sluggish refresh:
        howto blackList vga mode :
        sudo nano /etc/modprobe.d/blacklist-framebuffer.conf
        add the following line:
        blacklist vga16fb
        Note: you can enable and use remote access to a virtualbox machine...
        Enforcing static IP use for the virtual Ubuntu Server (192.168.0.78 in my case)

            sudo vi /etc/network/interfaces
            change
            auto eth0
            iface eth0 inet dhcp
            into
            auto eth0
            iface eth0 inet static
            address 192.168.0.78
            netmask 255.255.255.0
            network 192.168.0.0
            broadcast 192.168.0.255
            gateway 192.168.0.254
            store your dns info via
            sudo vi /etc/resolv.conf
            you should read/add something like this
            nameserver 208.67.220.220
            nameserver 208.67.222.222
             and finally
            sudo /etc/init.d/networking restart
            Install VirtualBox extensions on ubuntu servers
              apt-get install build-essential linux-headers-`uname -r`
              mount /dev/cdrom /media/cdrom
              cd /media/cdrom
              ./VBoxLinuxAdditions-x86.run
              Perform Ubuntu system update:
                sudo apt-get update
                sudo apt-get install ruby
                sudo apt-get install rubygems
                Install Ruby requirements for RedMine
                  sudo gem install
                  gem install rails -v 2.3.5 -no-doc -no-ri
                  gem install rack -v 1.0.1 -no-doc -no-ri
                  Plug Ruby with MySQL, using an adapter:
                    cd /tmp
                    check latest version http://github.com/tmtm/ruby-mysql/downloads
                    wget http://github.com/downloads/tmtm/ruby-mysql/ruby-mysql-2.9.3-beta.tar.gz
                    tar -xzvf ruby-mysql-2.9.3-beta.tar.gz
                    cd ruby-mysql-2.9.3-beta/
                    ruby setup.rb
                    Eventually install PHPMyAdmin
                      sudo wget http://sourceforge.net/projects/phpmyadmin/files%2FphpMyAdmin%2F3.3.7%2FphpMyAdmin-3.3.7-all-languages.tar.gz
                      sudo tar -xzvf 
                      sudo mv /apps/phpMyAdmin
                      On Ubuntu Server: symbolic links in /etc/apache2/sites-enabled point to 'available' web sites. Use the a2ensite (Apache2 Enable Site) command to create such symbolic links, like this: sudo a2ensite mynewsite where your site configuration file is /etc/apache2/sites-available/mynewsite . Similarly, the a2dissite utility is used to disable sites.
                      Enabling your PhpMyAdmin directory as a site
                        cd /etc/apache2/sites-available
                        ln /phpmyadmin /apps/phpmyadmin-3.3.7
                        sudo a2ensite phpmyadmin
                        /etc/init.d/apache2 reload
                        browse to address/phpmyadmin
                        login/root from installation credentials
                        sometimes php fails when triggered (security problems), edit
                        sudo nano /etc/php5/apache2/php.ini
                        and edit open_basedir = to add your own '/apps/' (which allow php execution from that directory).
                        Install Mercurial
                          sudo apt-get install mercurial
                          Download and configure Redmine (http://www.redmine.org/wiki/redmine/RedmineInstall)
                            chown -R www-data:www-data /apps/
                            cd /volume1/apps
                            wget http://rubyforge.org/frs/download.php/72201/redmine-1.0.1.tar.gz
                            tar -xzvf redmine-1.0.1.tar.gz

                            mv redmine-1.0.1 redmine
                            cd redmine/config
                            cp database.yml.sample database.yml
                            nano /config/database.yml

                            production
                            adapter: mysql
                            username: redmine
                            password: xxxxxxxx

                            sudo apt-get install libopenssl-ruby1.8
                            sudo apt-get install rake

                            cd redmine

                            RAILS_ENV=production rake config/initializers/session_store.rb
                            RAILS_ENV=production rake db:migrate
                            RAILS_ENV=production rake redmine:load_default_data
                            ruby script/server -p production

                            in redmine login/password: admin admin

                            Install passenger (module enabling execution of ruby servicing in Apache)
                              sudo apt-get install libapache2-mod-passenger
                              load module in sites apache config scripts if not included by default
                              LoadModule passenger_module modules/mod_passenger.so

                              Install RedMine site under Apache (http://www.he1ix.org/?p=572)
                                cd /etc/apache2/sites-available
                                Alias /redmine /apps/redmine-1.0.1/public
                                sudo a2ensite redmine
                                sudo /etc/init.d/apache2 reload
                                sudo chown -R www-data.www-data files log tmp public/plugin_assets
                                sudo a2enmod rewrite
                                cd /apps/redmine-1.0.1/public
                                copy dispatch.cgi.example dispatch.cgi
                                If you have problem with /tmp/mysql.sock (Redmine having some problems to access mysql: can't find /tmp/mysql.sock):
                                you cant use sudo  ln -s /var/run/mysqld/mysqld.sock /tmp/mysql.sock (file is not persistent), but you may specify the ubuntu mysql socket path in redmine config: edit /redmine/config/database.yml
                                production:
                                  adapter: mysql
                                  socket: /var/run/mysqld/mysqld.sock
                                
                                
                                Enabling NFS services on your synology: to allow your LAN machine to access a repository on the synology
                                  synoservice --list
                                  synoservice --enable nfs
                                  nano /etc/exports

                                  In 'exports' file, add the following line
                                  /directory/ ip.of.NFS.client(ro,root_squash,no_subtree_check)
                                  (* as 'ip.of.NFS.client', means 'for all computers on LAN')

                                  Save the 'exports' file

                                  Check if the file /var/lib/nfs/rmtab does exist, if not, run the following command
                                  touch /var/lib/nfs/rmtab

                                  Now perform
                                  cd / /usr/sbin/exportfs -a

                                  Testing: look at /var/log/messages to check whether the settings are ok with your Synology product

                                  On the Ubuntu Server side (read this):

                                  sudo apt-get install nfs-kernel-server
                                  sudo mkdir /apps/mnt
                                  sudo mount 192.168.0.77:/volume1/repos

                                  sudo vi /etc/fstab
                                  to add
                                  192.168.0.77:/volume1/repos /nfsmount nfs
                                  That's it for the virtual ubuntu server install !

                                    26 September 2010

                                    RedMine (Synology)

                                    RedMine is a great project/content management targeted at developers. I'm personnally using it as a complementary tool alongside Mercurial source code repository (cf previous posts):

                                    And after a terrible fight, i managed to get Redmine running on my Synology through Apache.

                                    First, Redmine is coded with Ruby, and that's partially a problem concerning the webserver to run this Web App. CGI interface is deprecated, and Mongrel, Thin, Unicorn, FastCGI gems (which are the usual suspects :)) are not built for our plateform. After spending some days trying, i resign myself using Webrick (which is not intended for production use, but for development, anyways it's working out of the box).

                                    Using Webrick, our ruby webapp will run standalone, accessible via http port 3000. If you want to use Apache to access it (because you cant access port 3000 from the outside, because you need secured transations) you'll need to set 'reverse proxying' to map one of your Apache web directory (https://diskstation/redmine/ ) to Webrick server.

                                    The most important about this installation is these references
                                    http://wiki.joachimschuster.de/index.php/Install_Ruby_on_Rails_and_Redmine_on_DS210%2B
                                    (choose the 'root install' way)
                                    http://www.vinc3nt.fr/2010/03/installer-redmine-sur-un-synology-ds209ii/
                                    (in french, use a dedicated redmine user which complicates things a little)
                                    http://www.redmine.org/wiki/redmine/RedmineInstall
                                    (Redmine official install howto)
                                    http://www.redmine.org/wiki/1/HowTo_Install_Redmine_in_a_sub-URI
                                    (and finally to connect the running instance with Apache)

                                    To write it again, the steps are:

                                      Setup MySQL (no other backend look usable right now)

                                        Activate MySQL services with Synology Console
                                        Create a 'redmine' database in MySQL (via phpmyadmin)
                                    http://forum.synology.com/wiki/index.php/How_to_manage_the_MySQL_database_using_phpMyAdmin
                                    (unzip phpmyadmin into shared web. copy phpmyadmin\config.sample.inc.php into config.inc.php, and add these ''. then flag it as read only.
                                    go on http://diskstation/phpmyadmin
                                      in privileges:
                                        create user in mysql db
                                        check 'create database for user + all credentials'
                                      Install 'Ruby and the gems'
                                    ipkg install rubygems
                                      Install the right version of Rake and Rails (long process => coffee time):
                                    gem install rails -v 2.3.5
                                    gem install rack -v 1.0.1
                                      Plug Ruby with MySQL, using an adapter:
                                    cd /tmp
                                    check http://github.com/tmtm/ruby-mysql/downloads
                                    wget http://github.com/downloads/tmtm/ruby-mysql/ruby-mysql-2.9.3-beta.tar.gz
                                    tar -xzvf ruby-mysql-2.9.3-beta.tar.gz
                                    cd ruby-mysql-2.9.3-beta/
                                    ruby setup.rb
                                      Download and configure Redmine (http://www.redmine.org/wiki/redmine/RedmineInstall)
                                    mkdir /volume1/apps/
                                    cd /volume1/apps
                                    wget http://rubyforge.org/frs/download.php/72201/redmine-1.0.1.tar.gz
                                    tar -xzvf redmine-1.0.1.tar.gz
                                    mv redmine-1.0.1 redmine
                                    chown -R nobody:users redmine/
                                    cp config/database.yml.sample config/database.yml
                                    nano config/database.yml
                                    (enter MySQL credential) 
                                      Session Key creation &  Database init
                                    RAILS_ENV=production rake config/initializers/session_store.rb
                                    RAILS_ENV=production rake db:migrate
                                      You should be able to launch Redmine from this point, with the command line.
                                    ruby /volume1/apps/redmine/script/server -e production
                                       Create a startup script, daemonizing redmine:
                                    nano /opt/etc/init.d/S97rubyrails.sh
                                    #!/bin/ash
                                      case "$1" in
                                       start)
                                         /opt/bin/ruby /volume1/rubyapps/redmine/script/server webrick -d -e production
                                       ;;
                                       stop)
                                         killall ruby
                                       ;;
                                       restart)
                                         $0 stop
                                         sleep 1
                                         $0 start
                                       ;;
                                       *)
                                         echo “usage: $0 { start | stop | restart}” &>2
                                         exit 1
                                       ;;
                                       esac
                                    don't forget to chmod 755 S97rubyrails.sh
                                      Now, you can access redmine via  http://synology:3000 .
                                    But Let's go one step further by implementing a proxy within Apache. The proxy thing helps if you need access to your redmine setup through the default syno apache frontend: it makes sense when you're enforcing security with passwords (Apache style), or only want to open your 80 port in your firewall.

                                      Finally, Reverse Proxying with Apache:

                                        Add this to /usr/syno/apache/conf/httpd.conf-user
                                    LoadModule proxy_module modules/mod_proxy.so
                                    LoadModule proxy_http_module  modules/mod_proxy_http.so
                                    ProxyRequests Off
                                    ProxyPreserveHost On
                                    ProxyPass /redmine/ http://127.0.0.1:3000/
                                    ProxyPassReverse /redmine/ http://127.0.0.1:3000/
                                    Apache server restart
                                    /usr/syno/etc.defaults/rc.d/S97apache-user.sh restart
                                      And finally the most important part: tell Redmine to prefix every url with a 'redmine/'when generating pages:
                                    Add the following line at the end of your Redmine config/environment.rb
                                    Redmine::Utils::relative_url_root = "/redmine"
                                      Done ! Hopefully. Access to redmine via http://synology.ip/redmine

                                      Final note: after some time on using RedMine, i wouldn't recommend using it on a low-end synology. As an example, my DS-107+ really takes age at updating project wiki pages. In fact, i ended using the NAS as a gateway between the outside, and some other private servers running on my private network... Next post should about this.

                                    20 September 2010

                                    Mercurial on Synology (hg web)



                                    I switched from a single Mercurial repository configuration (cf previous post) to one-per-project-so-multiple-repositories on my Synology.

                                    All my projects would be stored on NAS, located under a specific directory  /volume1/repos , and browsable through internet at http://diskstation/hg

                                    I was using ssh until now but someone suggested an hg web configuration: so, basically we are going to configure the native Apache web server to allow Mercurial transactions *and repositories browsing* via http/https.


                                    This is how i did it, the 'minimal Apache fuss' way, dealing with a dedicated web directory via .htaccess/.htpasswd files: it's far from perfect, security speaking, as it does not follow every Apache recommendations. 
                                    Sorry, but i didn't want to cripple this How-To with too much external considerations.

                                    1) Requirements:
                                    • being able to telnet into your synology box
                                    • being able to ipkg packages
                                    • having mercurial installed
                                    • having nano installed, or mastering vi
                                      (cf previous post)

                                    Note:
                                    • to edit text files, install nano with ipkg install nano
                                    • if you're editing files into windows through shared directories (using pspad ?) dont forget to save files the unix way (CR). Or Apache will fail @ parsing them.
                                    2) Enable Web Service on Synology:
                                          Apache web service should now be 'live', and you should have a new 'web' shared directory in /volume1/web or /var/services/web (both paths are identical).

                                    3) Creating a directory of Repositories (with correct credentials):
                                    • telnet into your diskstation, and type
                                        mkdir /volume1/repos/hg
                                        chown -R nobody:users /volume1/repos
                                      hgweb/apache would have problems to access repos otherwise (Error: abort: HTTP Error 500: Permission denied: .hg/store/lock), as processes are spawned with nobody:users credentials (check this with 'ps', looking at httpd processes group and user).
                                       could be good to create a shared repos/ directory (via syno control panel).
                                    • now create a /volume1/repos/hg/test repository, type :
                                         cd /volume1/repos/hg
                                         hg init test
                                         cd test
                                         nano test.txt (write some and save)
                                         hg add test.txt
                                         hg commit -u yourname -m first
                                    • create a /volume1/web/hg subdirectory
                                          cd /volume1/web
                                          mkdir hg
                                     
                                    4) Time for Script and Config 
                                    • create a .htaccess file to allow script executions in /volume1/web/hg/.htaccess
                                        cd /volume1/web/hg (or cd /var/services/web/hg)
                                        nano .htaccess

                                        (.htaccess is an Apache config file specifying options for this specific directory
                                    • insert this:
                                    AddHandler cgi-script .cgi
                                    Options +FollowSymLinks +ExecCGI

                                    AuthUserFile /var/services/web/hg/.htpasswd
                                    AuthName "Protected Access"
                                    AuthType Basic

                                    Require valid-user
                                      In this directory, Apache is allowed to execute cgi scripts and will protect access with login/passwords. If you don't want a password protected directory, erase all but the two first lines.
                                      Then, to password access /volume1/web/hg/:
                                    • create a .htpasswd file in /volume1/web/hg/ to store login info
                                        cd /volume1/web/hg    

                                        nano .htpasswd
                                      now, get and modify the Mercurial web cgi script (into /volume1/web/hg)
                                    • get mercurial hgweb.cgi from there or via
                                          cd /volume1/web/hg 
                                        wget http://www.selenic.com/repo/hg-stable/raw-file/7cf258b2d0cc/hgweb.cgi 
                                    • rename it into index.cgi
                                          mv hgweb.cgi index.cgi 
                                    • flag this python script as executable, and assign credentials (thx anon)
                                          chmod u+x index.cgi 
                                          chown -R nobody:users index.cgi
                                    • create a symbolic link to your favorite python executable: 
                                          ln /opt/bin/python2.6 /opt/bin/python
                                    • change /volume1/web/hg/index.cgi first line to: 
                                         #!/opt/bin/python
                                    • change in index.cgi (thx Mike):
                                    config = "/var/services/web/hg/hgweb.config"
                                    • create hgweb config file in /volume1/web/hg/hgweb.config
                                         nano /volume1/web/hg/hgweb.config
                                    • write
                                    [collections]
                                    /volume1/repos/hg = /volume1/repos/hg
                                    this config file is parsed by hgweb.cgi (now index.cgi), and will allow it to scan the whole directory to build the repository list. They are other options to play with.
                                    5?) Optionally, error feedbacks from Apache: 
                                    • edit apache config /usr/syno/apache/conf/httpd.conf-user
                                          nano /usr/syno/apache/conf/httpd.conf-user
                                    • to get log informations, change the line
                                          ErrorLog /dev/null
                                        into
                                          ErrorLog /var/log/httpd-error-user.log
                                    • save and restart apache
                                          /usr/syno/etc.defaults/rc.d/S97apache-user.sh restart
                                    • then, to access apache log
                                         cat /var/log/httpd-error-user.log|more

                                    Final notes: 
                                      Use the latter two to pull/push changes from/into other repos.
                                      Mercurial pushes are only allowed through https: it's possible to change this behavior in hgweb.config. Otherwise, dont forget to open your router to port 443 transactions.
                                      And yes it's working: i'm using it every day :)
                                      With Redmine. Next post should be totally about RedMine.

                                    References:




                                    31 May 2010

                                    XML Parsing and Indexing

                                    One of the idea behind GFE ([dje-fe], see previous post) is to be the less intrusive and the most reactive application. A front-end shouldnt need long processes stopping the user about to play, nor any private files written on disk (especially if you want it running from a DVD).
                                    But GFE needs to display informations about currently selected game (from a collection of 10000 items).
                                    Such data exists and is usually available through (big) xml files... They're generated by Mame.exe, or available from the Net (google 'dat file emulator').
                                    Hence the idea to parse like 40 Megs of xml data on the fly (at each run), and index/map the document to avoid keeping the whole file in memory.
                                    After two weeks studying the topic, it's time for conclusions :
                                    1. Microsoft Xml Pull Parser is fast. Written in .Net, but *correctly* written, it's generally a good tool for your xml needs. It's a pull system, easier to manipulate than SAX, and probably faster (look at XmlTextReader for reference). It takes 750 ms to walk 40 megs of data (8800 records) on my computer.
                                    2. Unfortunately you can't bookmark the interesting parts of your document with it. We could rely on the line/character pair this parser is returning but then another text parser should extract text blocks from line/characters pairs. Maybe tricky to implement and we'll lose some horsepower in the process.
                                    3. Note that parsing xml files is easy as long as we don't want document validation or xpath support: state machine only has to handle 10-12 token types, c# is silently handling various character format.
                                    So what ? Why not writing a dedicated indexing-xml-pull-parser ?
                                    Some results after a dozen hour of coding, a custom pull parsing implementation:
                                    1. It's now as fast as MS XmlTextReader:  first version wasn't :) too much function calls is hitting very hard c# performance (8 times slower than the original !), solution is to pack the whole state machine into a single function: now it's easily indexing  50 Megs of xml per second.
                                    2. It's able to give file stream position information about each element start and end: during some initialization pass, an indexer stores each record key and location values in a dictionnary, for further fast access.
                                    3. When GFE needs a particular record info, the dictionnary is requested for the location of the fragment which is read from huge file, and loaded  into memory to be thoroughly parsed. Apparently it's fast enough for 100 random requests per second. It's 100 times enough :)
                                    4. there is no 4.
                                    This 'works' and fits a particular situation for a particular embedded application: no memory stress, no alien file creation. Currently it's only limited by a minimal support of character types (it's only handling ANSI+unicode).
                                    But definitely it looks like an achievable way to go if you feel somewhat embarassed with existing implementation.
                                    Below is the code source of the main state machine function: contact me for more implementation details.
                                    public TokenType Next(TokenType notifs)
                                    {
                                    _insideElement:
                                        if (_currentType < TokenType.END_ELEMENT)
                                        {
                                            for (; ; )
                                            {
                                                //attribute
                                                switch (_ioBuffer[_bufferIndex])
                                                {
                                                    case ' ':
                                                    case '\t':
                                                    case '\r':
                                                    case '\n':
                                                        _bufferIndex++;
                                                        continue;
                                                    case '>': //end of start element
                                                        _bufferIndex++;
                                                        _depth++;
                                                        goto _beyondElement;
                                                    case '/': //empty element
                                                        _bufferIndex++;
                                                        if (_bufferIndex + 1 > _dataLen)
                                                            PrefetchIO(1);
                                                        if (_ioBuffer[_bufferIndex] == '>')
                                                        {
                                                            _bufferIndex++;
                                                            _tokenEndOffset = _bytesParsed + _bufferIndex;
                                                            _currentType = TokenType.END_ELEMENT; //EMPTY ELEMENT ! it starts and ends on the same token
                                                            if ((notifs & TokenType.END_ELEMENT) != 0)
                                                                return _currentType;
                                                            goto _beyondElement;
                                                        }
                                                        return DoExpected(">");
                                                    case '\0':
                                                        if (!FillIOBuffer())
                                                            return TokenType.END_OF_STREAM;
                                                        continue;
                                                    default:
                                                        _currentType = ProcessAttribute((notifs & TokenType.ATTRIBUTE) != 0);
                                                        if ((notifs & TokenType.ATTRIBUTE) != 0)
                                                            return _currentType;
                                                        continue;
                                    
                                                }
                                            }
                                        }
                                    //
                                    //beyond element: data, or comment, or pi, cdata
                                    _beyondElement:
                                        for (; ; )
                                        {
                                        _start:
                                            switch (_ioBuffer[_bufferIndex])
                                            {
                                                case ' ':
                                                case '\t':
                                                case '\r':
                                                case '\n':
                                                    ++_bufferIndex;
                                                    continue;
                                                case '\0':
                                                    if (!FillIOBuffer())
                                                        return (_currentType = DoEndOfStream());
                                                    continue;
                                                case '<':
                                                    _tokenStartOffset = _bytesParsed + _bufferIndex;
                                                    ++_bufferIndex;
                                                    for (; ; )
                                                    {
                                                        switch (_ioBuffer[_bufferIndex])
                                                        {
                                                            case '!': //comment, cdata, doctype
                                                                _bufferIndex++;
                                                                if (_bufferIndex + 7 > _dataLen)
                                                                    PrefetchIO(7);
                                                                if ((_ioBuffer[_bufferIndex] == '-') && (_ioBuffer[_bufferIndex + 1] == '-'))
                                                                {
                                                                    _bufferIndex += 2;
                                                                    ProcessComment();
                                                                    goto _start;
                                                                }
                                                                if ((_ioBuffer[_bufferIndex] == '[') && (_ioBuffer[_bufferIndex + 1] == 'C') && (_ioBuffer[_bufferIndex + 2] == 'D') &&
                                                                    (_ioBuffer[_bufferIndex + 3] == 'A') && (_ioBuffer[_bufferIndex + 4] == 'T') && (_ioBuffer[_bufferIndex + 5] == 'A') &&
                                                                    (_ioBuffer[_bufferIndex + 6] == '['))
                                                                {
                                                                    _bufferIndex += 7;
                                                                    ProcessCData();
                                                                    goto _start;
                                                                }
                                                                if ((_ioBuffer[_bufferIndex] == 'D') && (_ioBuffer[_bufferIndex + 1] == 'O') && (_ioBuffer[_bufferIndex + 2] == 'C') &&
                                                                    (_ioBuffer[_bufferIndex + 3] == 'T') && (_ioBuffer[_bufferIndex + 4] == 'Y') && (_ioBuffer[_bufferIndex + 5] == 'P') &&
                                                                    (_ioBuffer[_bufferIndex + 6] == 'E'))
                                                                {
                                                                    _bufferIndex += 7;
                                                                    ProcessDocType();
                                                                    goto _start;
                                                                }
                                                                return DoUnexpected("-");
                                                            case '?': //pi
                                                                _bufferIndex++;
                                                                ProcessPI();
                                                                goto _start;
                                                            case '\0':
                                                                //--- fill data
                                                                if (!FillIOBuffer())
                                                                    return (_currentType = TokenType.END_OF_STREAM);
                                                                continue;
                                                            case '/': //end element 
                                                                _bufferIndex++;
                                                                _currentType = ProcessEndElement((notifs & TokenType.END_ELEMENT) != 0);
                                                                if ((notifs & TokenType.END_ELEMENT) != 0)
                                                                    return _currentType;
                                                                goto _start;
                                                            default:  //new element
                                                                //(notifs & TokenType.START_ELEMENT) != 0);
                                                                _currentType = ProcessStartElement(true);//always in full as element name may is extremely usefull to get
                                                                if ((notifs & TokenType.START_ELEMENT) != 0)
                                                                    return _currentType;
                                                                goto _insideElement;
                                                        }
                                    
                                                    }
                                                default: //element data. (probably.)
                                                    _currentType = ProcessData((notifs & TokenType.DATA) != 0);
                                                    if ((notifs & TokenType.DATA) != 0)
                                                        return _currentType;
                                                    goto _start;
                                            }
                                        }
                                    }
                                    

                                    11 May 2010

                                    GFE: Game FrontEnd (emu frontend)

                                    Hey,

                                    I spent the last months fighting/juggling/coding with WPF. Not the easiest hobby in town, should i mention, anyway: this is GFE.

                                    The idea behind the project is to be able to... Play. No fuss, and more generally minimal settings, to be able to list and launch a vast collection of games: emulated, freeware, flash, choice is vast.

                                    Basically GFE is listing a directory of... things (roms, zip, pics, you tell), probably thousands of them.

                                    Background scanners are trying to match every entry with snapshots and movies (furtherly stats and extra infos from web). And apart from primary enumeration, everything else is asynchronous for a smoother experience.

                                    All settings are guessed from a directory structure passed as application argument or preferably from a gameinfo.xml file.

                                    It's using low level inputs, preferably a joystick, enabling some interaction with GFE running in the background. And it's a WPF project: so all the frontend rendering is done through DirectX and accelerated hardware if available.

                                    OK now, this is the very first version of GFE, and, well: it's usable, but probably not versatile or cute enough. I'm working on that: you can help with suggestions, money, or simple greetings.

                                    Binary:  GFE-0.5.0.zip (45 Ko)
                                    Technical details:
                                      Windows - .NET 3.5, works on XP and Seven
                                      Joystick (XInput compatible) or Keyboard
                                    How to run it:
                                      Modify included GameInfo.xml and use it as the application parameter.
                                    Note: when in background, both LB+RB button (Ctrl+Back) is killing launched app and bring back GFE to front. LB+Y exits GFE.

                                    alimbourg at gmail.com for any question.

                                    This is GFE Emu FrontEnd (yes it's all animated, and video is playing from the selected game):

                                    25 August 2009

                                    Mercurial on Synology

                                    First telnet, and boostrap to install ipkg, classic (look at this and uses this).

                                    Then ipkg install py26-mercurial (many mercurial flavours in repository btw: one for every major python version).

                                    Now mercurial should be installed somewhere in '/opt/bin' (which redirects to '/volume1/@optware/bin'), type hg -v to check this (or maybe hg-2.6 -v as ipkg apparently let me do the symbolic link: ln /opt/bin/hg /opt/bin/hg-2.6).

                                    You need to have some directory ready to hold a code repository, e.g. '//volume1/repos', executing hg init /volume1/repos/testhg will init/create a mercurial dir with (hidden) management files in it.

                                    We could start working in this, as a shared folder on the network (\\diskstation\repos\testhg), or via http (execute cd /volume1/repos/testhg, and hg serv, it mounts an http server on port 8000, accessible via 'http://192.168.0.X:8000' !), or via ssh (through 'ssh://user@192.168.0.x//volume1/repos/testhg') ! So Nice !

                                    I finally choosed the ssh way, to easily secure transaction.

                                    So, now ssh on synology:

                                    Let's activate the service using the web interface.
                                    Test it using kitty or plink with: plink.exe -ssh -2 root@192.168.0.X "set" should print a lot of variables contents.

                                    But, if we execute plink.exe -ssh -2 root@192.168.0.X "hg": it should fail with ash: hg: not found. When the ssh daemon starts a session, it uses only a subset of the usual user environment, so our PATH variable is missing directories, hence cant find hg.

                                    We might override this by editing (vi) /etc/ssh/sshd_config on the synology, to set PermitUserEnvironment yes.
                                    (to restart the ssh daemon after this, type killall sshd and /usr/syno/etc.defaults/rc.d/S95sshd.sh start)
                                    This script mod triggers the execution of a script $HOME/.ssh/environment for each ssh session creation: let's create such (executable) script with the line PATH=/opt/bin:/opt/sbin:$PATH in it.

                                    Now back to plink: we should now be able to execute hg through ssh.

                                    And... Everything should roll from this point :)

                                    (Once in a while, telnet into your box (putty) and ipkg update and ipkg upgrade)

                                    31 July 2009

                                    Obese debug information (sqlite and qtcreator)

                                    SQLite.c is the 'amalgemon' version of the Database: a huge .c file containing all the dependencies to compile SQLite into our projects.

                                    Using GCC, and standard mingw compilation option, we probably all notice the ton of compilation warning telling us that debug infos are too huge to be handled correctly (if you tried that, it's impossible to trace sqlite sources with gdb).
                                    The problem is concerning the embedded debugging information in STABS format: GCC is handling more modern types of debugging info. We just have to tell it to use them (http://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html)


                                    in your .pro file (assuming you're using QtCreator), force these flags to be used:

                                    #remove standard debug generation for c files
                                    QMAKE_CFLAGS_DEBUG -= -g
                                    #(*= 'if not present'), force uber debug infos, perfect for gdb
                                    QMAKE_CFLAGS_DEBUG *= -ggdb


                                    No more compilation warning=quickest compilation, and we're now able to trace sqlite.c in gdb... Another win ;-)

                                    Mame 132 and automated avi creation

                                    This is an How-To build a special mame.exe, wich allows some automation from AutoIt.

                                    I needed to find a way to send keys inputs which would trigger AVI recording of my games: but as Mame is using 'raw inputs', it cant be fooled with standard windows messages injection

                                    Fortunately the Mame project is really really (really) easy to modify and build, so i patched some sources to allow that.

                                    First, go there and follow every steps: http://mamedev.org/tools/

                                    Once you get your first build, modify some sources:

                                    Add More Windows Message to allow events from the outside:

                                    Augment raw inputs with incoming windows messages:

                                    Build it and voila. You should have a brand new mame.exe, listening for your external messages.
                                    I have such a binary build available for willing people.
                                    Next post should be an explanation for the whole Mame Avi Move Maker build with AutoIt...
                                    Regards (and eventually leave comments)

                                    30 July 2009

                                    Clutter 1.0.0: Win32/MinGW (updated)

                                    Personal Hack for a win32 build of clutter-1.0.0 libraries.
                                    • static libraries,
                                    • interactive.exe and the whole dll package (glib, pango, etc),
                                    • and a configuration package (prebuild.zip)
                                    In there.
                                    Basically all you need is:

                                    Get Code::Blocks/QtCreator whatever (but a Mingw flavor): http://www.codeblocks.org/downloads/5

                                    Get Clutter-1.0.0, and unzip-it somewhere on your hdd: http://www.clutter-project.org/sources/clutter/1.0/
                                    Get GTK dependencies from http://www.gtk.org/download-windows.html (binaries+dev packages): pango, cairo, glib, gtk+
                                    Get some other Mingw port from http://sourceforge.net/project/showfiles.php?group_id=7382 :
                                    gettext, libiconv

                                    I put all those libraries side by side with clutter-1.0.0.

                                    Concerning mingw, it's likely that the package delivered with cb/qtcreator is outdated:
                                    Get latest mingw API For MS Windows (http://sourceforge.net/projects/mingw/files/ , dev version) and unzip it into your mingw directory
                                    Get latest glext.h (http://www.opengl.org/registry/api/glext.h), to update your probably old \mingw\include\GL\GLext.h.

                                    Now some autoconf stuff:

                                    Get And Unzip prebuild.zip (http://alimbourg.free.fr/clutter-1.0.0) into clutter-1.0.0/build/
                                    It will create some qtcreator\ dir as it's my primary develoment platform, but rename it at will.
                                    The configuration package uses perl (i'm using tinyperl, included in archive) to autoconfigure clutter build, generate marshaling info and glib-enum-typing header files. It makes it ready to compile.
                                    Edit configure.pl to match your installation directories as it needs GLIB for some prebuild steps.
                                    Then, execute configure.bat.

                                    (Finally, it tweaks some qtcreator .pro project files to list every needed source files all the content from clutter\, clutter\COGL, clutter\Pango, clutter\json, clutter\win32: this, of course, has to be done 'by hand' for the other tools)

                                    It's now ready to compile...

                                    27 May 2009

                                    QtCreator Debug Helper...

                                    QtCreator (1.1 for now) is great tool to work with...

                                    The whole Qt thing is, too.

                                    I'm sharing some experience with the beast.

                                    If you need some relevant debug infos concerning the Qt class guts, you need to instruct gdb on how to print them.

                                    It's done via a 'debugging helper': some dll/plugin loaded by gdb on startup.

                                    On my machine they didnt work without a rebuild after each Qt SDK every integration:

                                    • go to Tools->Options...->Qt4->Qt Versions,
                                    • check the active Qt SDK, and its path (QT_PATH).
                                    • *remove* the whole *QT_PATH/qtc-debugging-helper* directory via explorer
                                    • back to the Options Pane, select the active SDK, and press *Rebuild*
                                    • go to Options->Debugger->Debugging Helper, check 'Use debugging helper', uncheck 'Use debugging helper from custom location'
                                    • next time you debug your qt app, you should get some '43 custom... loaded' from gdb

                                    13 April 2009

                                    Cool Dialog GUI (9 quads method)



                                    Hey all,

                                    If you want to draw some cool looking rectangle/dialog:


                                    this is the 9 quads method using... Clutter.

                                    First we need some texture like this (256x256 icon):














                                    To be loaded with:
                                    texture = cogl_texture_new_from_file  (texPath, 10, COGL_TEXTURE_NONE, COGL_PIXEL_FORMAT_ANY, NULL);


                                    Then, some layout parameters (inner rectangle coordinates):
                                    float framing[4]={108.0/256.0,50.0/256.0,200.0/256.0,147.0/256.0};


                                    and a function which splits the texture into 9 quads, to stretch the whole thing smartly:

                                    static void _cogl_9_rectangle (float x_1, float y_1, float x_2, float y_2,
                                    float framing[4], CoglHandle texHandle)
                                    {
                                    float  verts[12*9];
                                    
                                    if (!cogl_is_texture(texHandle))
                                    return;
                                    
                                    cogl_set_source_texture (texHandle);
                                    
                                    cogl_push_matrix ();
                                    cogl_translate (x_1, y_1, 0);
                                    
                                    float subrect[4];
                                    float w = x_2-x_1;
                                    float h = y_2-y_1;
                                    //uniformly stretched
                                    subrect[0]=w*dims[0];
                                    subrect[1]=h*dims[1];
                                    subrect[2]=w*dims[2];
                                    subrect[3]=h*dims[3];
                                    //and or preserving original texture dims
                                    int tex_w = cogl_texture_get_width(texHandle);
                                    int tex_h = cogl_texture_get_height(texHandle);
                                    float value = tex_w*dims[0];
                                    if (value<subrect[0]) subrect[0]=value;
                                    value = tex_h*dims[1];
                                    if (value<subrect[1]) subrect[1]=value;
                                    value = w - tex_w*(1.0f-dims[2]);
                                    if (value>subrect[2]) subrect[2]=value;
                                    value = h - tex_h*(1.0f-dims[3]);
                                    if (value>subrect[3]) subrect[3]=value;
                                    //and the 9 quads
                                    cogl_rectangle_with_texture_coords (0.0f, 0.0f, subrect[0], subrect[1],
                                    0.0f, 0.0f, dims[0], dims[1]);
                                    cogl_rectangle_with_texture_coords (subrect[0], 0.0f, subrect[2], subrect[1],
                                    dims[0], 0.0f, dims[2], dims[1]);
                                    cogl_rectangle_with_texture_coords (subrect[2], 0.0f, w, subrect[1],
                                    dims[2], 0.0f, 1.0f, dims[1]);
                                    
                                    cogl_rectangle_with_texture_coords (0.0f, subrect[1], subrect[0], subrect[3],
                                    0.0f, dims[1], dims[0], dims[3]);
                                    cogl_rectangle_with_texture_coords (subrect[0], subrect[1], subrect[2], subrect[3],
                                    dims[0], dims[1], dims[2], dims[3]);
                                    cogl_rectangle_with_texture_coords (subrect[2], subrect[1], w, subrect[3],
                                    dims[2], dims[1], 1.0f, dims[3]);
                                    
                                    cogl_rectangle_with_texture_coords (0.0f, subrect[3], subrect[0], h,
                                    0.0f, dims[3], dims[0], 1.0);
                                    cogl_rectangle_with_texture_coords (subrect[0], subrect[3], subrect[2], h,
                                    dims[0], dims[3], dims[2], 1.0);
                                    cogl_rectangle_with_texture_coords (subrect[2], subrect[3], w, h,
                                    dims[2], dims[3], 1.0f, 1.0);
                                    
                                    cogl_pop_matrix();
                                    }
                                    


                                    (http://code.google.com/p/syntaxhighlighter/ rocks)

                                    30 March 2009

                                    Clutter/Qt Creator IDE

                                    A new incomer into the free IDE arena: QT Creator from trolltech/nokia.



                                    The Windows version is based on the Mingw32 environment.... So Clutter is compilable with it.



                                    http://www.qtsoftware.com/downloads/sdk-windows-cpp


                                    Very neat IDE. Provided to spread Qt4.5 all around the world, this IDE is included in the Qt SDK. Dont be afraid, you can use it to build your project without any Qt in them.

                                    To build Clutter(*) on Windows using this one: you need to follow the prebuild.bat step (see previous posts), to allow Clutter to be built under a minimalistic MinGW environment.

                                    And get the project file Clutter.pro, for use with QtCreator, to be modified for your needs.

                                    cheers

                                    (*: Nokia/Qt and Intel/Clutter are competitors concerning mobile frameworks... Kinda ironic to build one using the other)

                                    20 March 2009

                                    Win32 Clutter (0.9.2 and Bleeding edge) compilation...

                                    ... Still using code::blocks:

                                    Okay i enhanced my scripts to make all this compiling with a native code::blocks install, but various clutter versions...

                                    Basically all you need is:

                                    - to get Clutter sources: using GIT repository (origin/master branch fe), or here : http://www.clutter-project.org/sources/clutter/0.9/ (0.9.2)
                                    - to create some %CLUTTER%\clutter\build\codeblocks directory
                                    - to unzip in there, content of http://alimbourg.googlepages.com/codeblocks.zip
                                    - to modify 'prebuild.bat' which configures various compilation defines, plus generates some .h/.c files using glib tools: so you have to change some paths to point your glib directory...
                                    - run 'prebuild.bat': it will ask for perl.exe, but if you don't have it installed, will copy my own "clutter-enum-types.c/.h" (hence, should be out of sync at some point)

                                    Read http://www.clutter-project.org/blog/?p=68 for dependencies. See previous posts about how to get them.

                                    After that, everything should be ok for you to compile...

                                    For Visual Studio Users: all you have to do is to create .vcproj with all the source files... But probably that some typical gcc include files would be missing (i dont have visual right right now to check that) like libintl.h, iconv.h (?).
                                    gettext, libiconv package for win32 are available here, but i dont know how they integrate themselves into a visual studio framework...

                                    PM or Comment if you need some more ;-).

                                    11 March 2009

                                    Clutter, 0.9 and above, places to look for

                                    BTW: for those interested in clutter 0.9.0 and above, i'm recommending using the test files as turorials as the other repo directories are still not very 0.9 friendly. And things did change a lot(!) between 0.8 and 0.9....

                                    Anyway, browsing clutter-xxx/tests/interactive and clutter-xxx/tests/conform directories should bring you with enough juice to start.

                                    Coming soon, some basic clutter code of mine.

                                    09 March 2009

                                    Bleeding Edge Clutter (Win32 build with C::B)


                                    Using GIT to retrieve repository:
                                    GIT binaries here: http://code.google.com/p/msysgit/

                                    • Start "Git GUI"
                                    • 'Clone Depot'
                                    • Source Depot is git://git.clutter-project.org/clutter
                                    • Target is yours: c:\Prog\clutter
                                    • Press 'Clone' and wait a while



                                    Thanks to them, they improved Win32 support :) No more dirty GL hacks to make it work.

                                    Get the latest CB files (see previous post about building Clutter on Win32): unzip this into your clutter\build\cb\: http://alimbourg.googlepages.com/cb-latest.zip

                                    Run prebuild.bat and that's it. Open Code::Blocks and the workspace: everything should rock'n'roll...

                                    I'm doing some application tests to confirm my first impression: Clutter *is* pretty smart (even if i miss the 3rd dimension of OpenGL :|)

                                    Cheers,

                                    05 March 2009

                                    Clutter 0.9 (CodeBlocks/MinGW/Win32 build)

                                    Do you know Clutter ?

                                    "Clutter is an open source software library for creating fast, visually rich and animated graphical user interfaces. "

                                    http://www.clutter-project.org/

                                    Great Project. Really.

                                    A bit too much intricated with gnome/gtk, not really win32 native ;), not so much pictures to display about, but... Nice and Smart code: really promising architecture.
                                    Wanted to figure this on win32, as i'd like to create some click n play frontends for my htpc...
                                    Good news: it's buildable on Windows/CodeBlocks (mingw) with a bit of luck and elbow grease :)

                                    Here is a recipe:

                                    Get Code::Blocks (Mingw flavor): http://www.codeblocks.org/downloads/5

                                    Get Clutter0.9, and unzip-it somewhere on your hdd.

                                    Get GTK dependencies from http://www.gtk.org/download-windows.html (binaries+dev packages): pango, cairo, glib, gtk+

                                    Get some Mingw port from http://sourceforge.net/project/showfiles.php?group_id=7382 :
                                    gettext, libiconv

                                    I put all those libraries side by side with clutter-0.9.

                                    Now some OpenGL trickeries: this is the main problem when compiling on win32 as internal OGL support is only up to 1.2, and clutter is more 1.4. So, as usual, we'll play with extensions and runtime function pointers retrieval.
                                    Note that clutter/cogl is dealing with >1.4 extensions as well: mine is a hack concerning 'some functions 'between' OGL 1.2 and 1.4... So:

                                    Get latest glext.h, to update your probably old \codeblocks\mingw\include\GL\GLext.h.

                                    Create clutter-0.9.0\build\cb directory ('cb' stands for code::blocks, took the directory location from the old msvc_2k5 port).

                                    Unzip this patch/project archive file in clutter-0.9.0\build\cb.

                                    Run cb/prebuild.bat: it will copy some config files, create the actor-marshall.h/.c, and replace some files with a patched version (have a look into prebuild.bat as it probably wont fit with your directory configuration).

                                    Now open code::blocks and cb\clutter.workspace: you should be ready to compile and link both clutter and interactive_d projects.

                                    To execute interactive_d.exe: you need a bunch of dll in the running directory as the content of clutter-0.9\test\data. This is my complete build folder.

                                    interactive_d.exe needs the name of the test to be ran, as a parameter, such as 'interactive_d.exe test-clutter-cairo-flowers'. Here is a two rows list of possible params:


                                    test-actor-clone test-actors
                                    test-animation test-behave
                                    test-binding-pool test-clip
                                    test-clutter-cairo-flowers test-cogl-multitexture
                                    test-cogl-offscreen test-cogl-primitives
                                    test-cogl-tex-convert test-cogl-tex-foreign
                                    test-cogl-tex-getset test-cogl-tex-polygon
                                    test-cogl-tex-tile test-depth
                                    test-easing test-events
                                    test-fbo test-fullscreen
                                    test-grab test-layout
                                    test-main test-model
                                    test-multistage test-offscreen
                                    test-paint-wrapper test-perspective
                                    test-pixmap test-project
                                    test-random-text test-rotate
                                    test-scale test-score
                                    test-script test-shader
                                    test-stage-read-pixels test-text-field
                                    test-text test-texture-quality
                                    test-textures test-threads
                                    test-unproject test-viewport

                                    That's it for Clutter 0.9 :)
                                    I'm going to dive more and more into it next days... And will publish my investigations in there. Stay tuned ?