The Coder's Way

Aug 08

Joining the team at Bislr in the down-under

Bislr

It is official I am moving to Australia. The down-under awaits me and my adventures are going to start tomorrow. I am going to be working at an amazing startup in Australia Bislr. I will be working on NodeJS and having fun learning new technologies. Bislr has an amazing product and I am excited to join their team to help contribute to it. I am so excited and anxious about my new venture. I am going to miss all my friends and family but with any transition, there is always growth and new opportunities. I want to thank everyone that has helped me to this place in my life and all the knowledge and their mentorship. I have created a new photo blog demetrius-downunder.com where I plan to share my personal adventure and odyssey. Please join me on this trip. I am sorry for the delay in my post, but I have been very busy with all my preparations to moving to Australia. When I get settled in Australia, I plan to start bloging again on both my technical blog and personal blog.

[video]

Jul 25

ZeroMQ Messaging Library Making Complex Communication Easy

I had been playing with ZeroMQ and I wanted to share it with you. First of all, it is not a complete messaging system such as RabbitMQ or ActiveMQ. A full fledged messaging system gives you an out of the box experience. ZeroMQ is not such a system at all; it is a simple messaging library to be used programmatically. It basically gives you a pimped socket interface allowing you to quickly build your own messaging system. This is very cool because you no longer need to use very complex monolithic systems, configure them, and have to have a dedicated system. It gives you sockets that carry whole messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fanout, pub-sub, task distribution, and request-reply. It’s fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems making it great for homogeneous environments. Specifically some benefits to using ZeroMQ are:

Quick note about running ZeroMQ on NodeJS. I tried to use npm to install ZeroMQ, and I had to make a few changes to the way ZeroMQ library was compiled on my mac OS X(10.6). You may have to change the warning level and install other libraries to get it working. Please contact me if you need help installing it!

ZeroMQ sockets are not the same as the sockets you know. When working with ZeroMQ, you need to think of them as magic communication hubs. Instead of thinking about the low-level socket and handshaking think about your message topology and flow. For example, you can have a ØMQ socket that binds to different ports and protocols at the same time. Your code will simply get the messages as if there is only one port and protocol. ZeroMQ also abstracts messaging patterns so if you want a ØMQ socket to act as a PUB/SUB or REQ/RES node you can just configure the ØMQ socket. Finally, ZeroMQ can inter connect and translate different protocols and messaging patterns so if you need to bridge a INPROC to TCP you can. Basically, ZeroMQ allows you to design your architecture and think about how you need all your systems to communicate with each other and not worry about how you are going to do it.

ZeroMQ handles all the complex work to make the messaging stable, fast, and reliable to let you focus on the fun part of architecting. The first abstraction in ZeroMQ is the transport layer. You can pick from a long list of transports. When designing your architecture ZeroMQ allows you to use 4 different transports:

The TCP transport is often the best choice, it is very performant and robust. However, when there is no need to cross the machine border look at the IPC or INPROC protocol to lower the latency even more. The MULTICAST transport can be interesting in special cases. Finally, after you have picked your transport layer you need to decide how you are going to wire up the message passers and receivers notice how I did not refer to them as clients and servers. This is because ZeroMQ can be used to message pass between processes on the same box or over the network without a central dedicated server. ZeroMQ is a low-level library that you link into your code that lets you build complex messaging systems. You have to decide what code is going to be the server or client and think about what parts of the architecture is stable and what part is elastic. This is apart of what makes ZeroMQ so cool you as a programmer get to build your messaging system but, you do not need to understand the complex nature of making a robust mature messaging system.

ZeroMQ has a lot of options on how to connect the message queues up. In a later post, I plan to cover some of the different type of queues, but there is already a lot of really good documentation on their API at www.zeromq.org. The key idea you should walk away with is do you want to send a message out to many workers to do processing or you want to fan-in all the request so you will know when all the work is done. Do you want to use a pipeline so you can have serial processing or do you want to do something else. You can achieve many different designs by picking the right messaging pattern. It is amazingly easy to setup a ZeroMQ queue that already has the logic and transport to support your requirements. So you can just start sending a message to ZeroMQ and let it do all the magic of sending the data and receiving it.

I have been playing with ZeroMQ to make a workflow system and a distributed processing system like Hadoop. I am loving it. ZeroMQ is really cool technology and has simplified my life a lot. Check it out, sometime when I have more time I will post some code examples!

Jul 09

Why Time and History is important to our Industry

A coworker and I were talking about the change in our industry and how it is under attack. He was very worried about how many disruptions are taking place like youtube, piracy, social marketing. He was frustrated that people do not understand that they are killing our industry and if we do not get paid for our content then they will never get any again. Specifically we were talking about the music industry, and I remembered a book I recently read about the history of the music industry. How short our memory is! The industry we talk about dyeing is the moguls like Sony, BMG, Universal, WEA, etc. The same guys that did the same thing to the live music venues a few decades a go.

That is right before the ability to record music the live venue’s where the king of music and held all the power. Then recording and radio displaced the live venues and became the new kings. Now the Internet has displaced the moguls again. So why is this bad? Did we need them to get music? Not really what is going on is a pattern I am seeing in the business world. The cost of making music, distributing, and marketing is dropping. Basically the market and social efficiency has been found and the industry is crying why they can not have the old days back. The pressure they feel is the pressure of their own lack of efficiency. Ether they find new efficiencies or change the business model or they will just die. This is the natural cycle in business. Nothing lasts for ever, and the evolution of change drives new winners and losers.  That is why they have turned to the government and the Law to protect them. Typically the role the government and Law is to stabilize transitions and slow change and growth.

Durning the growth period, recorded music and radio found new resources and exploited them. No longer where their limitations on the number of people you could pack into the live venue or the number of stars that could be featured on the weekend. The job of being a musician exploded and the rat race was on. At the beginning of this cycle cost of marking, distributing, and marketing music was so expensive they’re was a need for capital. If you think about the market place as a vacuum it pulls in all the needed resources to fill the void. Then everyone in the vacuum acts as a catalysts finding efficiency to fill it. But when you come close to the event horizon and find the maximum efficiency the ability to unfairly control and distribute resources ends. That is why the moguls are crying. At the beginning its easy to leverage and control the inefficiencies and profit off of it i.e. what the moguls do best.

Humans are amazing with their ability to find efficiency and exploit it. We also have a huge lust for acquiring stuff, physical or emotional. If you want to find the next big thing do not hang on to the end. Do not wait around, find the next cycle of birth so how do you do this? Its easy look for the same pattern. What is abundant and untapped? Like live venue they took a resource that was free the musicians that played for free singing in the fields and at home and created a market for them to play. The recording industry increased the market size and took the free resource i.e. the musicians and shared it with the people. Do you see the pattern? Find what is abundant and free that people already want and build a market with a different twist to it. This is the first step creating the vacuum then you need to become one of the catalysts finding the efficiency. If your lucky and one of the first to help fill the void you can profit off of it.

Everyone thinks startups are risky, that most don’t work, that they’re prone to failure. Actually, I find that working at a big company is what’s risky. Big companies tend to be more static; they don’t chase these new opportunities. They have a specific focus — they make turbines; they do television programming. They’re good at something and are very slow to adapt to new technology, while entrepreneurs and start-ups pick up on the new wave of technology and can be much more innovative.

What you’re involved in needs to scale, which means it needs to get cheaper every year and scale to millions of users. The company needs to be horizontal. You don’t do everything; you pick a slice that you’re good at. Like Intel — it doesn’t make computers; it just makes the chips that go into computers. And productivity is one of the most important things — it’s got to be productive. That doesn’t just mean being efficient; it means doing the right thing the right way.

Stay tuned I am going to keep writing about the pattern I see!

Jun 24

[video]

Jun 21

JavaScript everywhere one language to rule them all

We have JavaScript on the Browser, Mobile devices, Server, and now in the shell ;) what else can we ask for. JavaScript in our dreams? Maybe not but I really do love having one language to do all my work in. Not to long ago, I had to master a minimum of four languages to be productive. I would need to write my code in C#, JavaScript, PowerShell, SQL or PHP, JavaScript, BASH, SQL you get the point. Now with NodeJS I just need JavaScript to do it all. A few days ago, I needed to write a script that would create an account in the database, encrypt the password and link up all the permissions for that user across a few different tables. Then it would update some HTML based on rows added by the newly created user and push the code to the production server. It was easy as pie and came with all the benefits that the same code could run on my web server, command line, or cron job because its just a nodeJS package. In other words, its all JavaScript. I did not have to struggle to remember the crazy commands to do the work in BASH looking up man pages on sed/awk that I only use 4 times a year. It just is great to be able to use all the same tools you use every day. The same SQL client libraries, parsing tools, etc.

Writing a node shell script is easy it is no different from a Bash, Perl, or Python script. You just start the script with #!/usr/bin/env node and away you go! One hard thing with writing shell scripts is handling argument parsing and doing all the cool term outputs like progress bars, etc. You can just use packages like CLI or other NodeJS argument parsers to do all the heavy lifting so you can focus on writing code. I have started creating node shell script to do little tasks that I did manually. I am in love with my new workflow.

#!/usr/bin/env node

var cli = require('cli')
  , crypto = require("crypto")
  , dbclient = require('mysql').Client
  , db = new dbclient()
  , clId;

var options = cli.parse({
  host: ['host', 'DB Host', 'string', '0.0.0.0']
  , db: ['db', 'DB Database', 'string', 'database']
  , dbuser: ['dbu', 'DB username', 'string', 'admin']
  , dbpass: ['dbp', 'DB password', 'string', 'hummmmmm']
  , username: ['u', 'New Username', 'string']
  , password: ['p', 'Password', 'string']
});

var md5 = crypto.createHash('md5')
  .update(options.password).digest('hex');

db.user = options.dbuser;
db.password = options.dbpass;
db.host = options.host;
db.database = options.db;
db.connect();

db.query('INSERT INTO tableA (foo) VALUES(?)',
  [options.username], function(err, r) {

  if(err) return cli.fatal(err.message);
  cli.ok('INSERT INTO tableA >> ' + 
    (clId = r.insertId));

  db.query('INSERT INTO login (uid,pwd,clId) VALUES(?, ?, ?)',
    [options.username, md5, clId], function(err, r) {

    if(err) return cli.fatal(err.message);
    cli.ok('INSERT INTO login >> ' + options.username + 
      ','+ md5 +','+ clId);

    // do other crazy work

    db.end();
  });
});

JavaScript feels very good in a shell script with node supporting a REPL (Read Eval Print Loop) and its interpreted language it is a perfect fit. C# has PowerShell and Java has BeanShell but I just like the feel of nodeJS because the language is just a better fit.

Jun 16

Geospatial Search So Easy with Solr

For WeatherNation, I had to build an AJAX search box that would let users switch their current forecast zones. The requirements included searching citys, places, ICAO, zipcodes, and forecast zones. I also wanted to support additional signals for example if the user GeoIP was coming from colorado I wanted to rank Denver, CO over Denver, TN. This sounds like a great job for lucene or lets make it easier Solr. Solr is a wrapper around lucene and provides a very easy RESTFul interface. One of the hardest parts is setting up Solr so lets go do that now and get it out of the way.

$ cd /tmp/
$ wget http://apache.mesi.com.ar/lucene/solr/3.2.0/apache-solr-3.2.0.zip
$ unzip apache-solr-3.2.0.zip

$ mkdir /tmp/playground
$ cd /tmp/playground
$ cp /tmp/apache-solr-3.2.0/example/start.jar .
$ cp -rp /tmp/apache-solr-3.2.0/example/etc .
$ cp -rp /tmp/apache-solr-3.2.0/example/lib .
$ cp -rp /tmp/apache-solr-3.2.0/example/webapps .

$ mkdir -p data/weather/conf

At this point we have setup the most basic install of Solr. We are going to use start.jar that host the service in jetty and we have copied all the basic files need to run Solr. Lets update the basic configuration file. You can just replace the files with the version below.

# /tmp/playground/data/solr.xml
<?xml version="1.0" encoding="UTF-8" ?>
<solr persistent="false">
  <cores adminPath="/admin/cores">
    <core name="weather" instanceDir="weather" />
  </cores>
</solr>

# /tmp/playground/data/weather/conf/solrconfig.xml
<?xml version="1.0" encoding="UTF-8" ?>
<config>
 <luceneMatchVersion>LUCENE_32</luceneMatchVersion>
 <directoryFactory name="DirectoryFactory" class="${solr.directoryFactory:solr.StandardDirectoryFactory}"/>

 <indexDefaults>
  <useCompoundFile>false</useCompoundFile>
  <mergeFactor>10</mergeFactor>
  <ramBufferSizeMB>32</ramBufferSizeMB>
  <maxFieldLength>10000</maxFieldLength>
  <writeLockTimeout>1000</writeLockTimeout>
  <commitLockTimeout>10000</commitLockTimeout>
  <lockType>native</lockType>
 </indexDefaults>

 <mainIndex>
  <useCompoundFile>false</useCompoundFile>
  <ramBufferSizeMB>32</ramBufferSizeMB>
  <mergeFactor>10</mergeFactor>
  <unlockOnStartup>false</unlockOnStartup>
  <reopenReaders>true</reopenReaders>
  <infoStream file="INFOSTREAM.txt">false</infoStream> 
 </mainIndex>
  
 <updateHandler class="solr.DirectUpdateHandler2" />

 <requestDispatcher handleSelect="true" >
  <requestParsers enableRemoteStreaming="false" multipartUploadLimitInKB="2048" />
 </requestDispatcher>
  
 <requestHandler name="/update" class="solr.XmlUpdateRequestHandler" />
 <requestHandler name="/admin/" class="org.apache.solr.handler.admin.AdminHandlers" />

 <requestHandler name="/update/csv" class="solr.CSVRequestHandler" startup="lazy" />
 <requestHandler name="/update/json" class="solr.JsonUpdateRequestHandler" startup="lazy" />

 <requestHandler name="standard" class="solr.SearchHandler" default="true">
  <lst name="defaults">
   <str name="echoParams">explicit</str>
    <str name="defType">edismax</str>
     <str name="qf">
      name^1.8 locid^2.0 state^1.0
     </str>
     <str name="q.alt">*:*</str>
  </lst>
 </requestHandler>
 <admin>
  <defaultQuery>solr</defaultQuery>
 </admin>
</config>

# /tmp/playground/data/weather/conf/schema.xml
<?xml version="1.0" ?>
<schema name="weather lookup index" version="1.3">
<types>
 <fieldType name="integer" class="solr.TrieIntField" precisionStep="0" omitNorms="true" positionIncrementGap="0"/>
 <fieldtype name="string"  class="solr.StrField" sortMissingLast="true" omitNorms="true"/>
 <fieldType name="location" class="solr.LatLonType" subFieldSuffix="_coordinate"/>
 <fieldType name="tdouble" class="solr.TrieDoubleField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/>
 <fieldType name="text" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true">
  <analyzer type="index">
   <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
  <analyzer type="query">
   <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
 </fieldType>
</types>

<fields>
 <field name="name" type="text" indexed="true" stored="true" multiValued="false" required="true"/>
 <field name="state" type="text" indexed="true" stored="true" multiValued="false" /> 
 <field name="locid" type="text" indexed="true" stored="true" multiValued="false" /> 
 <field name="loc" type="string" indexed="false" stored="true" multiValued="false" /> 
 <field name="type" type="string" indexed="false" stored="true" multiValued="false" /> 
 <field name="rank" type="integer" indexed="true" stored="true" multiValued="false" /> 
 <field name="geoloc" type="location" indexed="true" stored="true"/>
 <dynamicField name="*_coordinate"  type="tdouble" indexed="true"  stored="false"/>
</fields>

<defaultSearchField>name</defaultSearchField>
<solrQueryParser defaultOperator="OR"/>
</schema>

Now that its installed and running lets test it.

$ cd /tmp/playground
$ java -Dsolr.solr.home=data -jar start.jar

Now Fire up your favorite browser i.e. Chrome just a suggestion ;). Go to http://localhost:8983/solr this is the admin interface to the solr service. Now that Solr is running we can load data into lucene. It is easy as pie, we can just import a csv file to be indexed. So create the file /tmp/city.csv see below. Next use curl to POST the city.csv to the RESTFul Solr service to have it load/index that file. This loads the cities  and latitude, longitude, zipcodes, etc.

# /tmp/city.csv
name,state,locid,loc,type,rank,geoloc
"woodland park",NJ,,"40.8892,-74.195",PLACE,4,"40.8892,-74.195"
Denver,NY,12421,"42.27,-74.53",ZIP,2,"42.27,-74.53"
Denver,PA,17517,"40.24,-76.13",ZIP,2,"40.24,-76.13"
Denver,NC,28037,"35.51,-81.01",ZIP,2,"35.51,-81.01"
Denver,IN,46926,"40.89,-86.05",ZIP,2,"40.89,-86.05"
Denver,IA,50622,"42.66,-92.33",ZIP,2,"42.66,-92.33"
Denver,MO,64441,"40.42,-94.29",ZIP,2,"40.42,-94.29"
"Denver City",TX,79323,"32.96,-102.82",ZIP,2,"32.96,-102.82"
Denver,CO,80201,"39.69,-105.08",ZIP,2,"39.69,-105.08"
Denver,CO,80202,"39.74,-104.99",ZIP,2,"39.74,-104.99"
"denver, centennial airport",CO,KAPA,"39.56389,-104.8483",ICAO,9,"39.56389,-104.8483"
"denver, denver international airport",CO,KDEN,"39.83278,-104.6575",ICAO,9,"39.83278,-104.6575"
"denver / stapleton international, co.",CO,KDNR,"39.78333,-104.8666",ICAO,9,"39.78333,-104.8666"
"denver nexrad",CO,KFTG,"39.78333,-104.55",ICAO,9,"39.78333,-104.55"
DENVER/ARTCC,CO,KZDV,"40.1833333333333,-105.133333333333",ICAO,9,"40.1833333333333,-105.133333333333"
"DENVER CITY 7W",TX,XDVS,"32.9833333333333,-102.933333333333",ICAO,9,"32.9833333333333,-102.933333333333"
"North Douglas County Below 6000 Feet/Denver/West Adams and Arapahoe Counties/East Broomfield County",CO,COZ040,"39.72,-104.80",ZONE,10,"39.72,-104.80"
"Denver International",CO,KDEN,"39.8333,-104.65",AIRPORT,5,"39.8333,-104.65"
"DENVER F. RANGE",CO,KFTG,"39.7833,-104.55",AIRPORT,5,"39.7833,-104.55"
DENVER/ARTCC,CO,KBJC,"40.1833,-105.133",AIRPORT,5,"40.1833,-105.133"
"DENVER WATER DEPARTMENT",CO,KBKF,"39.729,-105.009",AIRPORT,5,"39.729,-105.009"
"DENVER HEALTH",CO,KBKF,"39.727,-104.991",AIRPORT,5,"39.727,-104.991"
"DENVER FEDERAL CENTER HELISTOP",CO,KBJC,"39.723,-105.111",AIRPORT,5,"39.723,-105.111"
"DENVER POLICE DEPARTMENT-DISTRICT 3",CO,KAPA,"39.687,-104.96",AIRPORT,5,"39.687,-104.96"
"DENVER ARTCC",CO,KBJC,"40.187,-105.127",AIRPORT,5,"40.187,-105.127"
"DENVER OF THE EAST",NC,KIPJ,"35.493,-80.966",AIRPORT,5,"35.493,-80.966"
"DENVER CITY",TX,KGNC,"32.975,-102.842",AIRPORT,5,"32.975,-102.842"
"denver county",CO,,"39.76,-104.83",PLACE,4,"39.76,-104.83"
"denver nexrad",CO,,"39.78333,-104.55",PLACE,4,"39.78333,-104.55"
denver,CO,,"39.76800,-104.87270",PLACE,4,"39.76800,-104.87270"
denver,IA,,"42.67120,-92.33410",PLACE,4,"42.67120,-92.33410"
denver,IN,,"40.86420,-86.07640",PLACE,4,"40.86420,-86.07640"
denver,KY,,"37.77590,-82.85510",PLACE,4,"37.77590,-82.85510"
denver,MO,,"40.39900,-94.32370",PLACE,4,"40.39900,-94.32370"
denver,NC,,"35.53110,-81.03000",PLACE,4,"35.53110,-81.03000"
denver,NY,,"42.2125,-74.56972",PLACE,4,"42.2125,-74.56972"
denver,PA,,"40.23250,-76.13870",PLACE,4,"40.23250,-76.13870"
"denver city",TX,,"32.96960,-102.83070",PLACE,4,"32.96960,-102.83070"
Denver,IL,,"40.28330,-91.10000",PLACE,4,"40.28330,-91.10000"
Denver,NE,,"40.434655,-98.66677",PLACE,4,"40.434655,-98.66677"
Denver,TN,,"36.04694,-87.92083",PLACE,4,"36.04694,-87.92083"
$ cd /tmp/
$ curl http://localhost:8983/solr/weather/update/csv?commit=true --data-binary @city.csv -H 'Content-type:text/plain; charset=utf-8'

Next lets test the search by trying a few search results out.

http://localhost:8983/solr/weather/select?q=Denver

This will find all matches that have Denver. Our solrconfig.xml also lets Solr know that we want to boost fields using “name^1.8 locid^2.0 state^1.0” So the name field is more important than the state, but the zip code trumps all. For example if you had a city named “cool town” and a state “co” the “cool town” would be the first match then the state. Now lets try:

http://localhost:8983/solr/weather/select?q=Denver&sfield=geoloc&pt=40.08,-105.36&sort=score%20desc&bf=recip(geodist(),2,200,20)

This kicks in the Geo spatial magic, look at the fields sfield, pt, sort, bf fields. The sfield specifies which field is used to calculate the distance from pt. The pt is the point that all results will be measure against. In this example, (40.08,-105.36) is Boulder, CO next to Denver. The bf lets Solr know to boost the search results using the calculated distance from (40.08,-105.36) so Denver, CO wins over Denver, TX because Denver, CO is closer to Boulder, CO.

If you change (40.08,-105.36) to (32.78,-96.79) Dallas,TX you can see the results will boost Denver, TX to the top not Denver, CO. You can do so much more with Sorl, check out the documentation at Solr Spatial Search Wiki to learn more. I hope you find this useful, please leave comments if you have any questions.

Jun 12

Javascript Reflection & Introspection? huh

Javascript is such a cool language because it is so flexible. A friend challenged me to get JavaScript to support a feature that is commonly used in C#. In C# you can have custom attributes that can be applied to class, functions, arguments, packages, etc. This is very useful if you want to add metadata or declarative information to the code. I have built many things using this feature in C# like a REST API that uses the attributes to define documentation, ACLs, field validation, etc. This is very powerful because the code is very clean and declarative.

For exampl, lets say you want to make a generic REST API library so people can easily expose functions to be called throw a RESTful API. The standard way of doing this is to require the developer to explicitly add each function they want exposed to the REST interface by defining a configuration that maps a function to a RESTful endpoint. This setup and configuration code is separate from the code that has all the logic. Alternatively if you wanted a way to add metadata to the code that would not effect the way the function executes and stores the configuration details you could comment the code and use reflection to retrieve the metadata. C# and Java support this feature as a first class citizen but I do not know of a way to do this in JavaScript. So how can we do this? 

Let’s enter into the world of JavaScript and why I love it! JavaScript is interpreted at runtime so we can just ask for the code and parse out the metadata. So how do we do this its easy! I will walk you through the code below.

function parseMetaTags(metadata) {
 var tags = metadata.match(/\[@(.*?)\]/g)
  , restult = {};
 
 if(!tags) {
  return false;
 }
 
 for(var i in tags) {
  var str = tags[i]
   , len = str.indexOf('{')
   , tag = str.substring(str.indexOf('@') + 1,
       (len != -1 ? len : str.length - 1))
   , details = (len == -1 ? true :
       eval('(' + str.substring(len, str.length - 1) + ')'));
  restult[tag] = details;
 }
 
 return restult;
}

Function.prototype.reflect = exports.reflect = function(obj) {
 obj = obj || this;
 if(typeof obj === 'function') {
  if(obj.args) {
   return false;
  }
  
  var fnStr = obj.toString().replace(/[\n\t]/g,' ')
   , argStr = fnStr.match(/\(.+?\)/)
   , len = (argStr ? (argStr.index + argStr[0].length) : 11)
   , fnBodyStr = fnStr.substring(len)
   , metaData = {func:[], args:{}};

  if(argStr) {
   argStr = fnStr.substring(argStr.index + 1, len - 1);
   var argList = argStr.match(
         /(([$\s\w]+)?(\/\*.+?\*\/)?([$\s\w]+)?)+/g);

   for(var i in argList) {
    if(argList[i] != '') {
     var meta = argList[i].match(
         /(\/\*.+?\*\/)|([a-zA-Z_$][0-9a-zA-Z_$]*)/g);
     var key = false
      , val = [];
      
     for(var j in meta) {
      if(meta[j][0] == '/') {
       var details = parseMetaTags(meta[j]);
       if(details) {
        val.push(details);
       }
      }
      else key = meta[j];
     }
     metaData.args[key] = val;
    }
   }
  }

  if(fnBodyStr) {
   var bodyList = fnBodyStr.match(/(\/\*.+?\*\/)/g);
   for(var i in bodyList) {
    if(bodyList[i] != '') {
     var details = parseMetaTags(bodyList[i]);
     if(details) {
      metaData.func.push(details);
     }
    }
   }
  }
  
  obj.args = metaData.args;
  obj.func = metaData.func;
  
  return true;
  
 } else if(typeof obj === 'object') {
  for(var i in obj) {
   exports.reflect(obj[i]);
  }
  
  return true;
 }
 
 return false;
}

/* #### EXAMPLE #### */

function exampleAddUserToDB(
 arg1 /* [@REST{postName:'name', type:'string'}] */
 , arg2 /* [@REST{postName:'age', min:16, max:100}] */
) {
 /* [@REST{type:'POST', access:'public'}] */
 return true;
}

exampleAddUserToDB.reflect();
console.log(exampleAddUserToDB.args);
console.log(exampleAddUserToDB.func);

So what is going on in the code is very basic. The idea is we want to look for comments in the code that are formatted like /* [@attribute{json}]*/ and parse the comments out and turn them into metadata that can be attached to the function or its arguments. Because we want to add reflection to everything we add a prototype to Function. This makes it possible for any function to have reflect capabilities. The reflect function will call the toString() on itself. This gets the string representation of the function including all the comments so its just a matter of parsing out the tags in the code. After getting all the tags, we add the parsed tags onto the function itself so its easy to ask the function for its metadata. The function that parses the tags is parseMetaTags so you could build any kind of parser that would return any type of tags i.e. string, function, object, numbers. 

This lets you add any kind of metadata to your code via comments that follow a standard format. Than at any time, you can call reflect that will parse out all the metadata and return an object that your code can query against. Lets take an example of having a RESTFul service. We now can comment in our code to let the RESTFul controller query to find out if the function should be called using GET, POST, what the endpoint URL is, rules around required arguments, inline documentation that the RESTFul service can return, etc. Basicly anything you can dream up. When you call the exampleAddUserToDB.reflect() it parses out the metadata so it can be accessed by exampleAddUserToDB.args and exampleAddUserToDB.reflect.func. Below is the output of the example above. As you can see all the metadata has been pulled out of the code. You can now build any function you want and add metadata to it and use reflection to retrieve it. What is more amazing is this is not apart of the JavaScript language but we still can support it in just a few lines. That is what I love about JavaScript it is so powerful.

{ arg1: [ { REST: { postName: 'name', type: 'string' } } ],
  arg2: [ { REST: { postName: 'age', min: 16, max: 100 } } ] }
[ { REST: { type: 'POST', access: 'public' } } ]

I am not saying this is a good way to solve a problem but it is cool that we can do stuff like this. I love many things about C#, ObjC and Java each have their strengths and weaknesses but JavaScript seems to be more terse and flexible. This is both a good and bad thing! I really thing JavaScript is very very powerful, exciting, and fun to use and I hope sharing my weekend playground. I hope this with inspire you and your creativity to do other crazy cool things with JavaScript that it was not designed to do.

You can git (get) the code a github: demetriusj/js-reflection

[video]

Jun 03

Facebook to use Node.js!

Robert Kieffer, an a engineer at Facebook and the company’s internal Node.js evangelist, gave a talk at NodeConf today. He says the company isn’t doing enough with Node.js yet. So instead of an explanation of how to deploy Node.js to thousands of servers, he’s talked about why Facebook needs Node.js and the things holding the company back from adopting it widely.

You can find slides from this presentation here (PDF).

Some critics aren’t impressed because many of the problems Node solves can be solved with other technologies such as libev, Tornado, EventMachine, etc. So why should we be interested in Node.js? Kieffer because it breaks down the silos between developers. He says that at almost every place he’s worked, people are pigeon-holed into specific roles: front-end developer, backed-end devloper, etc. But that means that it’s hard for developers to pitch in to help other teams.

For example, Kieffer once worked for a company that was making an online document editor, back before Google Docs. He was the only front-end JavaScript developer on the project. A customer asked the company to build a Web-based calendar mockup. The CEO of the company, eager to please, decided that the team should actually build a full-featured app, not just a mock-up. Kieffer was stuck building the whole thing in a very short period of time by himself, because the other developers on the team weren’t able to write JavaScript. In a world where the same language was used for front, middle and back-end development, more people could have pitched in on the project.

So why isn’t Facebook widely embracing Node.js yet? Facebook prioritizes performance, as illustrated by HipHop. Even 1% improvement matters. But reliability is just as valued. Facebook has 600 million users - they will find any weakness in any new technology introduced. For a company like Facebook, there’s no one for its developers to look to as an example for how to do things. Also, rolling out new technology to thousands of servers is a non-trivial matter.

However, Facebook is using Node for a few things. One is JSGameBench, an HTML5 game benchmarker. Another is a mobile JS framework that has yet to be announced, but will probably be open-sourced. But the bulk of the Node.js work is being done by the Chat team for traffic analysis and load testing.

So how do you “cross the chasm” with an experimental technology like Node? Kieffer suggests an internal group, like the one he runs at Facebook. You can coordinate with other believers and get speakers to come in and enlighten others. You should also embrace your critics, he says, because they will point out problems with your ideas and keep you from screwing up early on. Finally, consider starting by applying Node.js to non-critical applications.