Contributing to Open Source is hard

I realized after trying to maintain the Go plugin for IntelliJ IDEA, how hard it is to actually contribute and maintain something for the open-source community.

Not only there's a problem with the time you have, my case after 10-12 hours of being away from home, getting at 8-9 PM to do something is hard. Then, there's the problem of switching from different context. Again, in my case, I'm switching from doing servers / automation or Go to Java and a whole bunch of different problems to solve, worlds apart would be a proper definition of it. Then you have to deal with issues coming in from your users, which contrary to what you'd expect, them being developers as well, 95% suck at reporting issues. And ofc they get impatient when their pretty little bug / feature request doesn't get attention from the developers, I guess it's easy to request things done for free by others in their spare time, no?

Don't get me wrong, occasionally, once a month or so, you get an e-mail from a random user saying: Thank you. (and not demanding anything else in turn) and that helps you remember why you are doing it as it feels so nice but more often then not people just don't understand how hard it is to do it.

As such, Thank you all of you out there who are doing such a nice work and share things with others, for free, in your spare time. And I don't mean those who get paid to do it from their job / as a living. If you open-source and maintain something that you did in your company, that's nice, but you got payed to do it. The real heroes are the ones doing it for free and demanding nothing in return.

How to build nginx from sources with some extra modules

Here's a short tutorial on how to build nginx from sources on Ubuntu. I've tested it on 12.04 and 14.04.

Pay attention to things like version numbers or dates and everything should work as a charm.

The life of a software developer

I've stumbled upon this: Working as a Software Developer but then I've realized that there's a couple of things missing from it:

- the number of programming languages one has to know and the way to solve problems in each of them. It's like speaking a foreign language for each programming language you know. And while you can learn the abstracts and apply them based on the language you need to write the code in, you still have to know the caveats and how to optimize things for the specific language

- the sheer size of the information in the industry one needs to keep up with every day / week / month so that you can always make the best decisions when implementing changes

How to cache responses in nginx and memcached

I've faced an interesting issue this week at work, how to improve our response times while not touching our software in the process.

Obviously, Varnish would be the answer for it but since our infrastructure architecture is already complex, there was a need to do it without introducing yet another layer just yet.

So, looking over what we can do, I've found out how nginx can cache things directly to memcached. Of course, it's always easy when you read about them but it turns out that there's no clear tutorial for how to actually do it.

As such, I've headed on the Internet trying to find a response to my question and quickly enough, there was a post about the same problem with no answer http://stackoverflow.com/questions/25639445/nginx-caching-response-from-remote-server-with-memcached/

Long story short, here's the solution I've came up with, maybe anyone else has a better one?

Add this line under nginx.conf (it adds support for Lua, see below why)

    lua_package_path '/usr/local/lib/lua/?.lua';

site config (in my case default):

    upstream memcached {

    server 127.0.0.1:11211;

    keepalive 32;

    }

    

    server {

    listen 8080 default_server;

    

    root /usr/share/nginx/html;

    index index.fhtml index.fhtm;

    

    # Make site accessible from http://localhost/

    server_name localhost;

    

    location = /memc {

    internal;

    

    memc_connect_timeout 100ms;

    memc_send_timeout 100ms;

    memc_read_timeout 100ms;

    memc_ignore_client_abort on;

    

    set $memc_key $arg_key;

    set $memc_exptime 300;

    

    memc_pass memcached;

    }

    

    location /memc-stats {

    add_header Content-Type text/plain;

    set $memc_cmd stats;

    memc_pass memcached;

    }

    

    location / {

    set_by_lua $key 'return ngx.md5(ngx.arg[1])' $request_uri;

     

    srcache_fetch GET /memc key=$key;

    srcache_methods GET;

    srcache_store_statuses 200 301 302;

    

    error_page 403 404 502 504 = @fallback;

    }

    

    location @fallback {

    proxy_pass http://127.0.0.1:80$request_uri;

    set_by_lua $key 'return ngx.md5(ngx.arg[1])' $request_uri;

    srcache_request_cache_control off;

    srcache_store PUT /memc key=$key;

    }

    

    }

My setup is like on Ubuntu 14.04, nginx running on port 8080 and Apache on 80 (just to test this) with nginx 1.7.5 compiled with the following arguments in "debian/rules" under "full_configure_flags"

    full_configure_flags := \

    $(common_configure_flags) \

    --with-http_addition_module \

    --with-http_dav_module \

    --with-http_geoip_module \

    --with-http_gzip_static_module \

    --with-http_image_filter_module \

    --with-http_secure_link_module \

    --with-http_spdy_module \

    --with-http_sub_module \

    --with-http_xslt_module \

    --with-mail \

    --with-mail_ssl_module \

    --with-http_ssl_module \

    --with-http_stub_status_module \

    --add-module=/opt/nginx/modules/ngx_devel_kit-0.2.19 \

    --add-module=/opt/nginx/modules/set-misc-nginx-module-0.26 \

    --add-module=/opt/nginx/modules/memc-nginx-module-0.15 \

    --add-module=/opt/nginx/modules/srcache-nginx-module-0.28 \

    --add-module=$(MODULESDIR)/headers-more-nginx-module \

    --add-module=$(MODULESDIR)/nginx-auth-pam \

    --add-module=$(MODULESDIR)/nginx-cache-purge \

    --add-module=$(MODULESDIR)/nginx-dav-ext-module \

    --add-module=$(MODULESDIR)/nginx-echo \

    --add-module=$(MODULESDIR)/nginx-http-push \

    --add-module=$(MODULESDIR)/nginx-lua \

    --add-module=$(MODULESDIR)/nginx-upload-progress \

    --add-module=$(MODULESDIR)/nginx-upstream-fair \

    --add-module=$(MODULESDIR)/ngx_http_substitutions_filter_module

I've compiled Lua and other modules, as you can see. The need for Lua was because I wanted to have a consistent way to hash the values for the memcached keys without having to worry about what will happen if someone will send some unexpected values as well as be able to hash it in the same way from the backend.


EDIT:
You can get the modules that I've added from here:
- Nginx Development Kit https://github.com/simpl/ngx_devel_kit
- ngx_set_misc https://github.com/openresty/set-misc-nginx-module
- ngx_memc https://github.com/openresty/memc-nginx-module
- ngx_srcache https://github.com/openresty/srcache-nginx-module

It's alive! (again)

The new blog is alive! (again). Now powered by Blogger.
Lets see how this goes.
I'll post more stuff #throughglass and Google Glass related soon.