Stritar's chronolog

Solving global and local records in SaaS database design

Sat, 20 Jul 2013 23:50:56 GMT

Lately, we've been seriously considering developing a new version of our framework. This one is already a few years old, and besides other problems it's facing, it lacks one very important thing - it's not suited for SaaS (Software as a Service) applications. Often, we would like to host a few simple projects (like multiple web pages) in a single database, but we are also thinking about developing a product / service, which we could offer to multiple clients. Making a product for different clients that would live in the same database is not simple, and requires an architecture that is both rigid and flexible, micro-useful and scalable.

The database model we currently use is doing quite well. It's able to support complex business-oriented systems, social portals and web sites, the core tables being the same across all applications. Sure, there are many redundant things, but we are aware where we should scale-down. More importantly, we have to decide how to upgrade this database model which is used in different applications to be able to work in a single database. I'm not that fond of using multiple databases or schemas in SaaS architecture, since this is a similar situation to which we have now, and it doesn't work that well. I'm certain having the same database for different clients is most suited for our business.

This means we will be having a ClientID (ApplicationID) in all the tables that will be used by all clients / applications. But what is the best way to take care of global and local records - records that are used by everyone versus those in only a few applications?

Let's assume we are building a CMS system for many sites, for which we will be needing two global Roles (Administrator, Moderator), but one instance will have its own Role as well (News editor). There are many ways we can go.

1. Local roles

This architecture means each client having it's own Roles. It's a simple solution, very flexible, but has many flaws - the most obvious being the benefits of global functionalities. E.g. if you're and Administrator, you can restart the site. Since we now have multiple Administrators (ID = 1,3,…), features like these get a bit harder to implement, and you can end up developing a totally different application for each client.

ApplicationID	RoleID (PK)	Role
1	1	Administrator
1	2	Moderator
2	3	Administrator
2	4	Moderator
2	5	News editor

2. Global Roles with a M:N table

The other option is to have global Roles, which are attached to a single application using a M:N table. This is an elegant and very flexible solution, but from my experience, you should avoid using M:N tables if possible. Of course, there are logical cases in which you can't, but you should always consider other options. Adding another table to the equation complicates queries, makes direct data browsing less understandable, besides raising the possibilities of errors.

RoleID (PK)	Role
1	Administrator
2	Moderator
3	News editor

ApplicationID (PK)	RoleID (PK)
1	1
1	2
2	1
2	2
2	3

3. Composite key

Another possibility that comes to mind is to use a composite key, which in reality, makes a single table from the above two (Roles, ApplicationsRoles). This allows using the same IDs for the same Roles in different systems. But it also requires having a composite key, which makes other things more complicated. It prevents making simple joins (you need to make a join by both primary keys), which means all the tables joined to Roles would need the ApplicationID field as well, even if they wouldn't really require it. Not to mention this model simply calls for trouble.

ApplicationID (PK)	RoleID (PK)	Role
1	1	Administrator
1	2	Moderator
2	1	Administrator
2	2	Moderator
2	3	News editor

4. Fake composite key

Sometimes, it makes sense using an additional primary key in a M:N relationship, so you can easily join the table further. We could do this as well, making the previous case a bit more easy-going. But this situation makes all the magic that we could do across the system a bit harder - since all the joins would be made using new local IDs, we are getting too close to the first solution (you can't make joins with the RoleID field, since it's not unique).

ID (PK)	ApplicationID	RoleID	Role
1	1	1	Administrator
2	1	2	Moderator
3	2	1	Administrator
4	2	2	Moderator
5	2	3	News editor

5. NULL-able ApplicationID

A different architecture enables using global Roles that have no ApplicationID, and local Roles that have it. Each application is set to use both sets of Roles, allowing flexibility and understandability. But this model has a problem as well - we can't prevent an application from seeing a global Role - those are intended for everybody, which means that in the long run, the system would start gravitating towards not having any global roles at all, to assure flexibility. To simplify - this architecture is great, but needs something that takes care of Denying.

ApplicationID	RoleID (PK)	Role
NULL	1	Administrator
NULL	2	Moderator
2	3	News editor

5. a) Creating a table for Deny

Let's put another application (ApplicationID = 3) into the equation, which, for some reason, doesn't want to have the global Role named Moderator (RoleID = 2). Adding another table to care of Denied Roles would solve the problem perfectly, but do we really need another table? We want to keep our database with as little tables as possible, so it's easier to use. That's why we should consider extending the Roles table to handle denying as well.

ApplicationID	RoleID (PK)	Role
NULL	1	Administrator
NULL	2	Moderator
2	3	News editor

ApplicationID (PK)	DeniedRoleID (PK)
3	2

5. b) Adding two fields for Deny

In the most normalized way, we need two fields to take care of Deny - a boolean to mark we are working with a Deny record type, and an integer to tell which record we are denying. But what if we want to optimize even further?

ApplicationID	RoleID (PK)	Role	IsDenied	DeniedRoleID
NULL	1	Administrator	false	NULL
NULL	2	Moderator	false	NULL
2	3	News editor	false	NULL
3	4	NULL	true	2

5. c) Using an existing field for Deny

We could put information about which record we are denying into the existing Roles.Role field, since we don't really need it in these cases. But there's a major setback; if a non-numeric field would be accidentally stored in the denied Role, the SQL engine would throw an error.

ApplicationID	RoleID (PK)	Role	IsDenied
NULL	1	Administrator	false
NULL	2	Moderator	false
2	3	News editor	false
3	4	2	true

5. d) Using a single field for Deny

It's much easier to combine the first two fields into one, marking the record implicitly. If the DeniedID is NULL, it means we are working with a regular record, otherwise, we are dealing with a denying record. Information about which Role we are denying is stored in the same field, which works both as a boolean, as an integer.

ApplicationID	RoleID (PK)	Role	DeniedRoleID
NULL	1	Administrator	NULL
NULL	2	Moderator	NULL
2	3	News editor	NULL
3	4	NULL	2

The decision

The last architecture (5. d) is the one I think it's best. It's very transparent and flexible, and it has only one major problem I can currently think of; if only a few (of many) applications would require to share a Role, this wouldn't be possible to solve with this model, but I can live with that.

ApplicationID	RoleID (PK)	Role	DeniedRoleID
NULL	1	Administrator	NULL
NULL	2	Moderator	NULL
2	3	News editor	NULL
3	4	NULL	2

The SQL statement

I have played around a bit with the 5. d) and it works very well. Here's the SELECT statement that retrieves the records for each specific application, supporting the mentioned scenarios.

SELECT Roles.*
FROM Roles
WHERE
(Roles.ApplicationID = 3 OR Roles.ApplicationID IS NULL) --retrieve local and global roles
AND
Roles.DeniedRoleID IS NULL --use only regular records
AND
Roles.RoleID NOT IN --remove denied roles
(
SELECT DeniedRoles.DeniedRoleID FROM Roles AS DeniedRoles
WHERE DeniedRoles.ApplicationID = 3
AND DeniedRoles.DeniedRoleID IS NOT NULL
)

The same query, solved with a JOIN instead of WHERE IN, which is faster, but probably less understandable.

SELECT RealRoles.*
FROM Roles AS RealRoles --regular records
LEFT OUTER JOIN
Roles AS DeniedRoles --denying records
ON RealRoles.RoleID = DeniedRoles.DeniedRoleID --join by the ID of the role
AND RealRoles.DeniedRoleID IS NULL --but only those records that are real
AND DeniedRoles.DeniedRoleID IS NOT NULL --vs those that are denying
AND DeniedRoles.ApplicationID = 3
WHERE
(RealRoles.ApplicationID = 3 OR RealRoles.ApplicationID IS NULL) --retrieve my and global roles
AND
RealRoles.DeniedRoleID IS NULL -- take all regular roles
AND
DeniedRoles.RoleID IS NULL --that don't have a deny

Conclusion

There are many ways to solve a database model in a SaaS architecture, but because of many reasons, we've decided to use a single database with an additional foreign key that defines the client in all the required tables. One of the biggest concerns in this architecture is solving global and local records, but the model presented here takes care of most requirements a typical Saas system might stumble upon, by using a NULL-able ApplicationID, with another field to take care of exceptions that require denying.

This architecture solves most of the problems, and if you're leaning towards a single database architecture, it's probably the best way to go.

Reinventing the blog, part 3: WTF did Pitchfork just do?

Thu, 30 May 2013 20:24:21 GMT

You know that feeling when you do a lot of research, and after you publish your results, you notice something else that should be included as well? Happens to me all the time. Immediately after creating the list of the most outstanding blogs, I noticed a few others that should be added to the directory. Shit happens. I also tried to identify the funky new UI elements these blogs use, which would help me understand the user experience requirements driving modern publishing trends. Felt pretty good about the conclusions, but only to find myself out of luck again. Turns out I missed something very important, something I noticed when I saw what Pitchfork does with their cover articles.

When I was doing some research on Daft Punk, I read an article on Pitchfork, and their use of parallax scrolling almost brought tears to my eyes. Not that parallax scrolling is anything new - there are many outstanding cases of using this technique in web design. But usually, this means the whole website is a single-page showcase that is using multiple layers that scroll with different velocity and in different directions to display information in an interesting way. One page.

Not on Pitchfork. They're using advanced layouts and super-fancy parallax scrolling on all of their cover articles. Custom fucking everything. Contrary to some other magazines, who are also using generic parallax scrolling in their feature articles (respect!), and some other blogs using custom layouts for cover stories (respect!), this is simply over the top. Check it out for yourself. I can't imagine how much time is put into a post like this, making it looks as good as it does.

One of the crazy transitions and animations inside the story.

But what does this mean? Simple. Do you remember The Daily, an iPad-only magazine that was already discontinued? Supposedly it offered revolutionary ways of consuming content. I didn't get the chance to try it out, but I did saw a few issues of Wired on the iPad, and I have to admit, it did feel like being in the future.

These paradigms are slowly being adopted on the Web as well, and some magazines and blogs are already polishing their most important content to impressive levels, combining in-depth stories and custom development into digital masterpieces. Like Pitchfork's cover story, or even a feature on Newsweek or a review on The Verge. This goes beyond putting some text into a WYSIWYG editor and uploading a few photos. Modern (digital) publishing is obviously pointing towards a collaborative effort of a broad team of journalists, designers and web developers, which will be the only way to deliver content that will retain audience. So much for a plain and simple blog redesign.

Check out the complete Reinventing the blog series.

Reinventing the blog, part 2: The challenges, the opportunies

Thu, 09 May 2013 12:42:06 GMT

After the initial dissection of the most innovative and well designed blogs, it's time to go behind the scenes. We've seen how some of these blogs look like, but there's even more value in understanding why they look like they do. Every good user experience analysis needs to have a clear overview of the goals and good insight into the problems of the situation, and I will try to outline these by using my blog as an example. A lot can be deducted by monitoring the basic Google Analytics reports.

Even though generic blog challenges and opportunities may not exist, I think that most blogs probably share around 80% of these goals. And just to be clear, by blogs I don't mean corporate blogs with their specific requirements (sales, leads, conversions), but modern online magazines, originally (and still) called blogs, even though they are actually becoming mainstream media. Therefore, this analysis can work for most online publishers, but I'm doing it mostly to help me understand what I need to do to reinvent my blog again. It's pretty simple: getting and retaining traffic. All other requirement are derived from these two.

Main goal: getting traffic, increasing engagement

The main goal of every website is to get traffic. For traffic, you need great (or at least good) content, something with added value for the users. What you do with that traffic, is another story. Sell things, sell ads, sell yourself.

There are primarily three different sources of traffic you can receive: direct, referral and search. You want all of them, and each component is a result of different activities. Getting direct traffic means you have a strong brand. Getting referral traffic means you have great writers / influencers which help you get backlinks to your blog. Getting search traffic means you have a great technical team and SEO (CMO and SMO) optimized site.

You want a well-balanced stream of traffic to your blog.

The other important goal that websites have is keeping the traffic on the site, which can be done by stimulating user engagement. Helping things happen after you get people to your blog. You want users to spend as much time as possible with you, navigate through many pages, and hopefully give you some feedback - leaving comments and sharing your posts on social media. Some blogs want other actions as well, but these are probably the most obvious ones.

Increasing social media activity

Social media is emerging as one of the most important referral sources. But lately, it's becoming a noteworthy factor for search engine rankings as well. That's why blogs are trying hard to integrate social media widgets into their content, some even go beyond the standard tweet / like / + 1 buttons. Help people share, there are many benefits from it, both direct and indirect.

The social activity on my most visited post. These figures translate into more than 10k unique users.

Lowering bounce rates, increasing time on site

Most blogs have very high bounce rates - people tend to read a single post and leave the site. That is why designers and information architects are trying to do everything to keep readers on their site, hoping to persuade them to read another article. Some are trying too hard by adding too many related posts, others are playing with many interesting ways of driving readers to the next page.

My current bounce rate is more than 80%, with average visit duration just above 2 minutes. Something that needs to be improved urgently. I am sure this will be one of the most significant challenges I will face when developing the next generation of my blog.

Improving these bounce rates and time on site is one of my highest priorities.

Treating every page as a landing page

The time when people visited the homepage and navigated further from there are over. Today, specially because of social media, every page is a landing page and needs to be treated as such. We are consuming content in a different way, we don't browse for content anymore, the content finds us. That is why we need to design every single page as the starting point for our visitors.

At this point, less than 10% of traffic to my blog comes through the homepage. Which means designing great post pages is becoming more important than designing the homepage, since every page needs to become a hub for further navigation.

Building loyalty and credibility (branding)

Besides keeping users on site, it's also important you get users back to the site. As your brand and reputation evolve, it gets easier with all of the traffic components, but you can achieve that with only one thing - great content. Of course, nice design and user experience help, but you can't bypass this simple fact: you need something with added value to the readers. Only then you will get loyal users and only then your blog will thrive.

Even though most focus is done on gaining new readers, you mustn't forget about returning ones.

Allowing a mobile-friendly experience

We are living in a mobile world and more and more traffic is made by mobile devices. This number is closing in on 20% on my blog, which means optimization for mobile visitors is becoming something that you can't ignore.

Mobile operating systems are on the rise.

Combining it with the funky new UI elements

In my previous post, I outlined the interesting new user interface elements my favourite blogs use. Now we can map these elements with the above mentioned five challenges and opportunities, which will help us understand the requirements behind the innovations.

	Social media	Bounce / time	Landing pages	Branding	Mobile
Unconventional navigation		1	1	1
Fixed (floating) menu		1	1
Responsive design		1		1	1
Big key visual before text	1	1
Advanced key visual (gallery, video)	1	1		1
Abstract, teaser, subheading	1				1
Suggested content within limits		1	1		1
Custom social media integration	1			1
Polished content (wrapping, quotes)	1	1		1
Custom typography				1
Infinite scroll		1			1
Pinterest-style homepage		1		1
Integrated content and navigation		1	1	1
Total	5	10	4	8	4

It's pretty clear each of the elements has at least one specific goal that it's trying to solve. Some of them will surely become a standard in the future.

Conclusion

What a modern blog is trying to achieve in 2013 is pretty clear. Get more traffic, make users spend more time on the site. This hasn't changed in decades, but the behaviour of the users has. Only those that will be able to adapt to the newly-formed situation will survive.

The most important paradigms that have shaped the past years are the social and mobile era, which have revolutionized the ways we consume content. We are online all the time, with many devices, and there is so much content everywhere, it's overwhelming. Our attention span is becoming very limited, and all of above mentioned challenges are focused in trying to capture it.

I decided it's time to redesing my blog again, and by understanding what (and why) the market leaders do, my task will surely be easier. I hope you find my analysis useful, but please feel free to add anything I may have forgotten in the comments.

Check out the complete Reinventing the blog series.

Reinventing the blog, part 1: Dissecting the most innovative and best-designed blogs

Fri, 29 Mar 2013 15:23:46 GMT

The blog is getting mature. Ever since Gawker did its eccentric redesign a few years ago, we've seen a lot of other blog (networks) doing similar things, trying to reinvent how the blog should look like in 2013. After the initial hiccup, Gawker managed to fortify its position and attract new users, showing others that people do like to see different things, things that are imitating the experience of reading electronic magazines on mobile devices. Today, there are many great cases of how a modern blog should feel, and since I'm thinking about doing something similar myself (it's been almost 4 years since I did this!), I decided to dissect a few of the most innovative ones, hoping to get a picture of what works and what not. Here are my picks of the most creative and best designed (mainstream) blogs on the Web, those that are standing out from the crowd and are unique in what they offer to their readers.

Fast Company

Fast Company is a great example that demonstrates the power of images - especially if you have access to professional photographic material. The home page is clean and the highlighted article is integrated into the main key visual, which works very well. The same logic of the huge picture is implemented to the inner pages as well, and this picture is transformed into a gallery if required. There isn't much of related content on a single article, and the social buttons are custom, which I think we will be seeing a lot of in the future.

Fast Company uses an effective combination of the key visual, article header, additional flavor text and custom social buttons.

Fast Company uses custom typography, infinite scroll to avoid paging, takes good advantage from the article abstract / subtitle to make you curious (it's displayed on the article as the introduction as well), and their website is responsive. It is also interesting that they keep a single article visible above the fold on their homepage to retain focus. From the design and user experience perspective, this solution is one of my favorites.

Gawker

Gawker's innovative approach was to use two columns for news on the home page - one for top stories and one for latest stories - and ditching the main menu. While this may have been one of the most important evolutions modern blogs have made, I don't think this feature works well on the homepage, since I don't really notice the right column when I browse the site. However, this feature becomes more useful on the inner pages, where this module is duplicated, and where most people land on the site.

Gawker and the omnipresent main menu that changed the game.

This realization, that single blog posts should be treated as primary landing pages, is very important, and Gawker was one of the first to fully integrate that concept into its user experience. People don't browse blogs anymore, they consume social media that brings them to blogs.

Gawker also uses a big picture (not in such a cool way as Fast Company) in the post, and it's very smart and concise that this picture can be replaced with a video. I like the way comments are solved, showing only the most popular threads, and not the complete conversation (with the amount of comments they have, it would be probably useless otherwise). I'm also keen on the internal hot meter they use, which they seem to use to distinguish the top and latest news. However, they should ditch the "like Gawker" block exposed on each article, it's very misleading. The mobile site should also be replaced with a responsive version.

Mashable

Mashable's redesign introduced quite a few interesting features I look forward to adopting. They have minimized the main menu, offering a drop down popup menu that allows further classification of news. This menu is well coded and does a pretty good job of taking care of incidental mouse moves (not as good as Amazon though). The homepage uses three columns to display articles, even though I'm not fully sure how that works ("The new stuff" is probably all articles, "The next big thing" are probably highlighted by the editor, and "What's hot" by the crowd), and this feature's column header is fixed upon (infinite) scrolling. The design is, driven by their specific social media ninja audience, of course, responsive.

Mashable has a clever integration of social media activity on the top of the article.

What I like about Mashable's new version the most, is the clever social media integration. They have the total number of shares displayed on the top of the article, as well as a little graph that displays the dynamics of social activity for a specific post. These social media statistics are probably also used to feed the columns on the homepage, even though most people probably don't understand what's happening. But perhaps that's for the best - if it works well in recommending the articles, thumbs up.

Again, we are seeing a picture above the article, which can be video as well. That's good. What I don't like about Mashable is the three-column footer of the article, it is the same as the homepage, displaying a single category. I can understand the need for such a thing, it could work, cloning the homepage on the landing article page, but for me, it's just too overwhelming.

The Next Web

The Next Web's new design is somewhere in between Mashable and Gawker. The homepage uses two columns, the Channels, which can be configured and switched from popular to latest, and the main window, which offers a pinterest-style display of articles, similar to the one Mashable uses. This left menu box is fixed and used both on the home and inner pages, and it works as the main menu to navigate the content of the portal. The main main menu is simple and works as a hub for other TNW stuff.

The Next Web's navigation and post header, together with instruction to use the keyboard.

The Next Web also has custom social media integration, and offers the users to navigate with the keyboard. This navigation works very well with the left box - meaning the users is navigating the current selection in the box, offering an experience similar to switching a remote on a digital TV, knowing what the next channel will be. I'm not sure how many users notice and use this feature, but this integrated content and navigation approach is very interesting.

The site is responsive, uses a big picture before the post, together with the detailed information about the article. Another interesting thing - the images break out of the paragraph form. Overall, a very solid performance with a minimalistic design.

Quartz

While Quartz may not be one of the most well-known blogs out there, it still very much deserves a mention for its creative implementation. The clever menu that collapses when you proceed to the article, the interesting fixed list on the left that can be configured and pivoted according to your wishes, making the navigation much easier and again, integrated with the content.

Quartz fully integrates the navigation and content.

There's another innovative feature on Quartz for which I haven't decided if I like it or not, but it is very interesting nevertheless. When you scroll to the end of the article, the next article is automatically displayed, together with a new URL. This is made as a redirect (the url changes), but it does not seem like one at all - when I get the time I will try to see how they technically achieve this. This article change is integrated with the left box, which makes the complete experience pretty interesting, similar to the one The Next Web has. What Quartz misses is better social media integration. They went a step back and decided to use links to share pages instead of widgets, which probably doesn't help their traffic that much, but it's aligned with the design. The site is responsive.

The Verge

The Verge's homepage is probably the most unique of them all. While I would make the main menu less confusing, I really like the tiles for the most interesting articles. This is followed by the video section, and by a ton of other articles - too many of them, to be honest. But things get more interesting once you get to a specific post. The menu gets smaller, there is a clear navigation to the next and previous articles at the top, and the breaking news floats.

The Verge's posts are close to perfection - full of images and quotes, together with embedded galleries.

The design of the inner page is very creative, there is a huge image (not on all articles, it seems only on the reviews) with social share widgets, a secondary title is also used. I really like the quotes inside the text, and the jump-to thingie is useful as well. You can see that someone took a lot of time to shape the content, the text is wrapped around images, the galleries are embedded within the text. This gives you an impression you are browsing a high-end iPad magazine rather than a web page. There aren't to many other elements on the page, so the overall result is very clean and easy to read. On the other hand, that polished content structure probably makes it quite hard for the site to be responsive, which The Verge is not.

Wired

The legendary Wired magazine has an online edition that would make a lot of people proud. They have been always known as innovators, and were one of the first to introduce the grid display of articles instead of a list. This means the emphasis is more on the images and headings than it is on the text. The popup menu is put somewhere inside this grid of posts, which is a daring, but effective solution. This menu neatly moves to the top on the inner pages.

Wired's article headings have big titles and teasing abstracts.

Again, we are seeing two different types of posts as on The Verge, the basic one, and the advanced one. The advanced one is a feast to the eyes, with a huge heading and abstract that get you interested, the pictures that break out of paragraphs. But there are a few things that are not suited for such an established magazine. In a gallery, each click reloads the complete page, which can be very very annoying. I don't think hunting for ad views makes it worth it. I would also make the right column a little less overwhelming with content (not only ads, but everything else as well). And the site is not responsive.

Breaking down the elements

Based on the analysis, we can conclude that new specific elements started to emerge with the next generation of blogs. These elements are the results of us consuming content in a different way that we were a few years ago - before social and mobile. Most of them are taking care of "the homepage is not the landing page" situation, while trying to persuade people to proceed with browsing the content, lowering bounce rates.

	Fastco	Gawker	Mashable	TNW	Quartz	Verge	Wired	Total
Unconventional navigation		1		1	1			3
Fixed (floating) menu			1	1	1			3
Responsive design	1		1	1	1			4
Big key visual before text	1	1	1	1	1	1	1	7
Advanced key visual (gallery, video)	1	1	1					3
Abstract, teaser, subheading	1	1				1	1	4
Suggested content within limits	1	1		1	1	1		5
Custom social media integration	1		1	1				3
Polished content (wrapping, quotes)			1			1	1	3
Custom typography	1		1	1	1	1	1	6
Infinite scroll	1		1		1			3
Pinterest-style homepage			1	1		1	1	4
Integrated content and navigation				1	1			2
Total	8	5	9	9	8	6	5

Conclusion

Typical design and user experience of mainstream blogs have evolved in the past few years, and we will be seeing similar concepts adopted by the mainstream media as well. The social and mobile era have changed the way we consume content, while heightening our expectations - most of us simply count on great experiences. Luckily, the world is full of great innovators who are not afraid to take risks and implement new creative features that will become a standard in the years to come. I'm already looking forward to how other major players will respond to the new situation.

Check out the complete Reinventing the blog series.

The Risk board game dice roll probability calculator and battle simulator

Sun, 17 Mar 2013 16:32:12 GMT

I know there are plenty of you out there who love to play the board game Risk. We're hooked on the Lord Of The Rings edition, and I still need to check out the very rare expansion pack one of my friends recently got. As you will see, I'm getting ready for it with all I've got, developing myself a weapon that will help me dominate the game. Something that will turn the odds in my favor without actually cheating. Say hi to my Risk battle simulator, which is able to calculate the chance of winning for specific Risk situations.

Launch

Risk is an interesting game, powered by simple mathematics. In the battle, the attacker throws three dice, the defender throws two. The attacker's advantage is the one extra die. The defender's advantage is that he/she wins when the dice are tied. Which, in the general situation of 3 dice vs. 2 dice roll, translates into the following odds: the attacker has 37.17% chance of winning, the defender has 33.58% chance of winning, and there's a 29.26 % chance of tie - they will both lose one army. But what about specific situations? How can I know if my 10 armies are enough against my opponent's 5+4?

Since I'm a developer and not a mathematician, I decided I will rather build a simple brute force JavaScript simulator than try to derive the formula behind the battles. An application that simulates 10.000 Risk fights (around 50.000 dice rolls) and calculates the result odds. That should be enough to make an approximation, right?

Risk's rules specify that the attacker always has to leave one army behind (can't use it to attack), and can attack any number of territories in a single turn. Which the calculator also takes into account. Every territory the attacker wins, his/her number of armies is subtracted by 1 (one army is left behind), and the last army is not able to attack. I think I got it right, but if you find and error, please let me know.

Risk dice roll battle simulation.

Besides, the simulator also knows how to calculate probabilities for specific single battle situations for different numbers of dice the attacker and defender throw, by going through all the possibilities of dice throw results (which means 7.776 (6⁵) different dice throws in 3 vs. 2 battle).

Risk dice roll possibilities and odds.

I made this to help myself and others understand and appreciate the statistics behind the dice. So, the next time you play Risk, don't forget to bring your phone and use the simulator to your unfair advantage. And if you're a developer, feel free to upgrade the algorithm (available on Github). Game on.

http://diceroll.stritar.net/risk.html
https://github.com/gstritar/DiceRoll

I believe Firefox OS may be on to something

Sat, 02 Mar 2013 22:27:50 GMT

A few days ago, during the Mobile World Congress, Firefox announced its mobile OS, which will be available soon. Teaming up with 18 carriers and 4 announced manufacturers (plus Sony), the release was probably bigger than expected. A few high-profile web services, including AirBnb, Disney, Facebook, SoundCloud and Twitter, also joined the hype by including their apps to the new marketplace. Analysts quickly put down their bets, some supporting the effort, while others denying the possibility of its success. One of the most fascinating things about the new OS is that it's going to be entirely web based, the operating system itself, the apps, everything. Unlocking the power of the web, as they put it. And to be honest, I can buy that.

The situation

Currently, the mobile OS market is dominated by two players, Apple and Google. They both have their own strategies, Apple being the control-freak offering exclusivity, and Google being the easy-going dude appealing to the masses. Windows is trying to find its place somewhere in-between, but it's still struggling to gain its market share (currently at around few percent) - we will see how their partnership with Nokia turns out in the long run. We mustn't also forget about Blackberry and their potential comeback with their new operating system and the newly introduced Z10 smartphone. But that's about it.

Until Firefox OS was introduced.

The history

To begin with, we must take a look at the original Firefox browser, the first-choice browser of the developer a few years ago. That is before Chrome managed to offer a stabler and faster version of it. What made Firefox so useful, were the Javascript console and Firebug, an add-on that all web developers need once they try out. But Firebug supposedly makes Firefox work much slower. Chrome built such a tool for client-side debugging inside its browsers and boom - millions of developers switched to Chrome. I don't know why Firefox hasn't offered a similar tool, they should, especially now, when they have a chance not only to gain mobile OS market share, but also to regain their position in the browser wars. They should fully unlock the power of the web, with their potential mobile OS and browser marketing synergies. Firefox = the internet.

The web is wonderful, and Firefox has always been one of its strongest advocates. And now they are doing it again, by offering an operating system that is fully based on the web. Besides, for many reasons, they are probably in a much better position to do it then WebOS was. Or as they put it:

With Firefox OS, you can simply enter any search term and instantly create a one-time use or downloadable app.

The community

Developers are an important part of every mobile ecosystem, and developing for different platforms is a big pain in the ass. Of course, everybody has the possibility to decide for HTML app instead of a native app, but if the platform prefers native apps, it's a no brainer that those will have more capabilities and better performance. But it's hard to make native apps. I am a web developer, been doing it for years, tried to develop something for iOS one day. I lost interest in a few days, because you need to get used to a totally new environment, and the thought of going through the same with Android and Windows just made me depressed. But here's what Firefox says:

Every Web developer can easily create and distribute HTML5 apps so you can find an app for whatever you want.

Holy crap, the web as the platform! Which means I will be able to make Firefox OS apps already from the start. And when I have that HTML5 app, will I perhaps be willing to easily turn it into a hybrid native / HTML5 app for all other platforms? The thought is appealing. And since HTML5 is powerful enough to access the phone's hardware (camera, GPS, etc.), this makes it much more interesting. Not to mention I would be improving my basic web developer skills if I would start developing for Firefox OS.

The recipe

Firefox OS has the carriers, manufacturers and supporters aboard, and if developing and deploying apps will be as easy peasy as they brag about it, it will all come down to a single thing: the interface and user experience that the rendering engine will be able to provide. Firefox has its own rendering engine Gecko, contrary to WebKit, which powers Chrome, Safari and Opera. Based on the first videos of the Firefox OS, it seems the interface isn't as smooth as the one you can get from the iPhone, Android or Windows Phone, and not as innovative as Ubuntu Touch. Which can be a big, big problem. Manufacturers may need and support alternatives, but it's the users who will decide, and their expectations are very high.

If Firefox can take care of this, and (stronger) devices are fully adjusted to run it, while developers are able to upgrade the user experience, I don't see a reason why Firefox OS shouldn't gain traction. All the components are there, the brand is strong and the race is long. Go Firefox!

5 reasons why I won't steal your idea

Tue, 05 Feb 2013 09:22:39 GMT

Since I'm a software architect and a web developer, I get often approached by people with their new ideas. In most cases, for some quality feedback, and on lucky days, for a rough quote about the costs of such a project. These people are usually very secretive about what they have, making me explain to them that it's far from my interest to steal that idea. One time, a guy even made me sign a Non-disclosure agreement before I could make him an offer for a service he was thinking about. After bargaining with me, he chose a different contractor, but ended up doing nothing, at least to my knowledge. He was obviously focused on the wrong things, instead of getting feedback from as many sources as possible, he was investing energy into bureaucracy and protection of his idea. Let me tell something to him and all others out there: Focus on your product, and don't worry about me stealing your idea. I won't. I have at least five reasons not to.

1. Your idea probably isn't as great as you think

I've seen a lot of different people who had "game-changing" ideas, at least so they though. A few of them actually managed to convince me and my partners that their idea is so amazing that it'll kick everyone's ass. Even though proper market research wasn't done, charisma is sometimes hard to resist, and if you are working with someone you've known for a long time, you are prepared to accept crazy terms, such as a delay of payment until this idea will start to generate revenue. After these specific ideas were put into the real world, it turned out there is a huge discrepancy between ideals and reality, and we ended up with unpaid invoices and ignored phone calls.

Ideas are something, execution is everything else. There is a long way inbetween, a way paved with upgrades, downgrades, changes, pivots, time and hard work. Millions have ideas, only a few can make them work. I've seen Seedcamp companies change their core concepts and business models completely, and these startups are already the best, selected from hundreds, if not thousands. When you start working on something and proceed ahead, the initial idea will often evolve beyond recognition. Not to mention there is a very strong possibility that someone else was already thinking about the same thing, except better, years earlier. Your idea isn't amazing, but it may be good enough to achieve something with proper execution. That's why you need feedback and partners.

2. I have plenty of ideas of my own

You know what's better than your idea? My idea!

Mostly because I kick ass, but also because people get emotionally attached to the thoughts they generate by themselves. I have so many ideas I don't know what to do with them. They are probably not really great (see reason 1), but they are mine, and I try hard to make a few of them come alive every now and then, when I have the time. During the day, I work on real-life projects, during the night, I play around.

When I decide on what to work on next, I usually look for the best ratio between actuality, complexity, required energy and potential. This means I've already made my own priority list of the services I will be rolling out in the future, and I must say, it would really be hard to put one of yours inside this packed list. I'm sure most developers think in a similar fashion, lacking resources to make everything they imagine a reality. Face it, there are hundred times as many people who have unrealized ideas, than people who don't know what to work on. Do the math.

3. Your idea probably requires specific passion and know-how

The idea lives strongest in the person who thought of it. It is a result of that person's experience from many fields, so it's hard to replicate in its full form without that experience. The ideas I've stumbled upon usually solve very specific and niche problems you can't solve without digging yourself into that field. Which most of us don't have time or the resources to do. The core of the idea represents the person who thought of it, it may be taken to another level by a different person, but in most cases, it requires the original author's knowledge, involvement and passion to work as it's supposed to.

I can't execute an idea which will revolutionize kindergarten children education, I don't know shit about the problem, I don't have any connections in the industry, and I'm simply not that passionate about that field.

4. Your idea requires your involvement as a product manager

In the past few years, we've implemented quite a few prototypes and services together with Neolab, and even though a few of them got some praise, they didn't make it to the mainstream. Take this blog for example, I developed it in early 2009, aggregating posts from different social services, presenting them (also) in a magazine form. What did I do with it? Nothing. Years later, a service called RebelMouse did something similar and raised millions in funding. Same goes for Twenity, a spin-off from Twitfluence, gamifying social authority measuring. Or Ljubljana Realtime, a social event discovery tool. All out there, but that's it.

I simply don't have enough energy to push a service beyond a point, or don't want to. Perhaps this fact will change someday, but at this point, you will need to be the product manager of your idea, and I can be its architect. Since I have a real company to run besides all of this, I can't afford to be one.

(Btw, if you think you could do anything with the above mentioned things, don't hesitate to give me a shout).

5. Karma and stuff

I believe in karma. Don't do evil and all of that. I would really feel uncomfortable if I would take someone else's baby and make it my own. So I won't, because this simply wouldn't be a fair thing to do. I value proper sleep above success.

But what if

Of course, there are always exceptions, and I can imagine I could encounter something that would go beyond all of my points above. Perhaps there is one idea that I've heard about years ago that would suit this description. If I ever decide to proceed with this project, I will let that person know what I'm doing and invite him to join the project. Even if I'm thinking about a thing that only faintly resembles the original concept, I can't deny it's that person's idea. And since this guy was able to think of such a marvelous thing so much time ago, he would surely make a great addition to the team (also see reason 3).

That's it. I won't steal your idea, so feel free to talk about your revolutionary innovation with me anytime. All I will do is to try to tear it apart and put it back together, and after we're done with that, if I get the chance, I will try to bring it to life.

Using JSON (with asp.net) is like wiping your ass with silk

Sat, 05 Jan 2013 18:45:44 GMT

I love mashups. Actually, I love everything about them, I love using them, I love making them, I love those who do everything they can to empower them. In my opinion, mashups are one of the most significant concepts the Web has invented, since they represent unlimited possibilities of integrating and reshaping things that are already done. The platforms out there are stable, so it's the creativity that sets the limits. These days, you can easily take data from anyone and do something else with it. Just don't forget to use JSON.

I'm an old school guy who has been working mostly in enterprise software environments, which means using XML for data integrations, besides, RSS used to be the man. The first generation of Twenity (Twitfluence) was using XML feeds, and this blog is using mostly RSS for fetching entries from other sources. XML was the standard we all spoke, very cute and readable, but on the other hand, not so easy to parse. At that time, this fact didn't represent such a major problem, since a few more lines of code took care of everything. But today is a different situation.

It was Twitter who first started dropping support for XML, which annoyed the hell out of me. I had to start rewriting things for JSON. This turned out to be one of the best things that ever happened, since JSON seems to be loved by everyone. It's super easy to use with jQuery, but since I'm a server-side type of a guy, it's even more important that it's cleverly integrated into asp.net. Fetch the data, store it, do crazy shit with it.

JSON is serializable into a .net object with a single line of code. Create the class with parameters compliant with the the specific JSON structure, serialize the response string into that class, and everything automagically works. Piece of cake, unlimited opportunities. Take Twitter for example:

Make the basic Tweet class:

public class Tweet
{
public string id_str;
public string text;
}

Create the request to access a tweet (funny, the hardest thing to do):

string url = "http://api.twitter.com/1/statuses/show/274508827146215424.json";
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Method = "GET";
WebResponse response = request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string jsonResponse = reader.ReadToEnd();
reader.Close();

Which will return something like this (play here):

{
"created_at": "Fri Nov 30 13:42:59 +0000 2012",
"id_str": "274508827146215424",
"text": ""Facebook knows what we say, Google knows what we think",
"source": "web",
...
}

Serialize the response string into an object:

JavaScriptSerializer js = new JavaScriptSerializer();
Tweet tweet = new Tweet();
tweet = js.Deserialize<Tweet>(jsonResponse);
Response.Write(tweet.text);

Pretty neat. Since I've started using JSON, mashups have become easier than ever to make. With one of our latest projects, we've integrated our application with Twitter, Instagram, Foursquare and Flickr. For breakfast! XML may have its advantages, but for such things, JSON is simply the greatest. All hail the new lord!

Hey developer, here's something that will make you sound smart

Thu, 13 Dec 2012 08:21:02 GMT

I've met many developers in my life, and quite a few of them share a similar problem. Being mathematical geniuses and all, but not being able to put into words what the hell they are doing. At least so it would sound marketable and awesome. After all, it's not their job to sound smart, the developer's role in the Hipster - Hustler - Hacker dream team is a bit different. But talking like an MBA can have it's advantages, specially when it comes to individuals communicating with their clients.

Launch

Take my example, half of the time my customers don't fully understand what I'm saying even though I'm trying really hard. But I've noticed some phrases have a better effect than others, some simply sound like special awesome things are happening (which they are) and that everything is under control. The funny thing is that the recipe to speak like that is very simple: say hi to the MBA developer talk.

[MBA verb] + [technical noun] = [#mbadev]

The equation for the #mbadev talk is elementary. Take a power verb that has a really active meaning. Like "structuring", "evaluating" or "utilizing". These are often used to make an activity sound way cooler than it actually is. Then, take a noun that is very hacker-specific, something that non-technical people don't fully understand. Like "metadata", "framework" or "encapsulation". Put them together, and you have a winner. Developers, face it, no one really understands what you're saying, so you might as well make it sound cool.

"Dear client, I'm very busy structuring metadata, thank you for your understanding".

The MBA developer talk seems such a great concept that we've even made a generator for it. Feel free to take it for a spin, you'll be amazed by how such a basic combination can yield such interesting results. Now all developers can sound really smart.

http://mbadev.neolab.si

A case study in agile development: the algorithm for Ljubljana Realtime's event discovery

Mon, 22 Oct 2012 20:01:16 GMT

When we started working on Ljubljana Realtime, we decided to approach it in an agile way. Amongst others, we wanted to use a few interesting lean concepts such as rapid development, Minimum Viable Product and the Build - Measure - Learn iterations. Less than two months later, the results are in, and they are very pleasing. The MVP in the shape of an activity map was developed in a few weeks, only to show there is a lot of social noise which will somehow need to be taken under control. But that's currently low priority, since the first pivot is already taking place, slowly shifting the focus from the rich map application towards an event discovery algorithm and stream. That's where I see the most potential of Ljubljana Realtime, and in the last weeks, that's where the most work was done.

Launch

Ljubljana Realtime event discovery engine uses Foursquare trending venues and geo-tagged posts from Twitter, Instagram and Flickr to discover what's happening in real life. At least 6 people checked-in on Foursquare or two different people tweeting or posting photos in a single hour could mean something is going on. These events are posted to Twitter and Facebook, with links to the posts. A few versions of this algorithm were already deployed, each one solving new problems, resulting in a few micro Build - Measure - Learn cycles in a single month.

Iteration 1: Foursquare, no duplicates

The first version of the stream (bot) was a simple one, at that point it was meant to work as promotion for the map. The only thing it knew how to do was to wait a few hours until it posted the same thing again. I think Foursquare checkins are alive for three hours, so if a trending venue was still trending after that time, new people had to checkin and the venue was still buzzing.

Problem: Plain, no real added value.

Iteration 2: Adding activity from other sources

When we were trying to make some space on the crowded map, we started grouping posts from Twitter and Instagram by the nearest Foursquare venue, which meant having less boxes on the screen. This turned out to be quite a complex thing to do properly, but it was worth the effort. On only a few occasions, one venue would have multiple posts in a single hour, and in most cases, that meant something was happening there. This provided another very interesting potential for the activity stream. Actually, it made the stream bigger than the map could ever be.

(I love it when such things happen, when you are trying to solve a problem, and it turns out there is much more hidden behind the resolution.)

Groupping posts by a venue. Did Ljubljana Realtime just discover an athletic meeting taking place?

The next problem: Activity in some venues, specially generic ones such as "Ljubljana" would trigger the stream almost every day. Similarly, some large venues, such as supermarkets, would be trending too many times on Foursquare.

Iteration 3: Balancing the posts

The algorithm needed an update, which would lower the amount of times when a venue would be recognized as an event, either on Foursquare or on other channels. At first I though about an upgrade which would set the amount of people or tweets needed to trigger the "event discovered" action for a specific venue. This would enable us to reduce the importance of some venues, but it would also require manual work. Luckily, we came up with another brilliant idea: the more times a venue is trending, the harder it is for it to be trending again, at least for the next few days. Automatic balancing.

Venues with the most discovered events. Generic ones, besides massive places, such as train stations, cinemas, squares and shopping centers are too dominating.

The next problem: At this point, we have launched other test instances of Ljubljana Realtime (Maribor, Zagreb and Zurich), to see how the system behaves in other environments. Some cities are bigger, some are smaller, which means they produce different amount of activity. Besides, different services are used differently in different cultures.

Iteration 4: Supporting local instances

Foursquare is big in Croatia (Zagreb), but not so much in Switzerland (Zurich), which means Zagreb Realtime's stream had a lot of Foursquare trending posts, while Zurich's had a lot of "Increased activity on Twitter and Instagram" posts. It was obvious that local instances needed different algorithms. While having an option to set the amounts which would trigger the post on a specific venue would be too much to moderate, having the same logic on a specific region could work. And it does. Zagreb now needs more people checked-in on Foursquare, while Zurich needs more unique people tweeting or sharing photos.

Number of discovered events by type (Foursquare vs. Twitter + Instagram) on each day. Foursquare trending venues are dominating Zagreb, while social streams are dominating Zurich Realtime.

The next problem: The basic algorithm requires two different people to tweet/post from the same location in one hour. In case of Zurich, this amount was set to three, but it turns out this situation happens rarely, around 10 times fewer than with two people, or only two to three times a day. Obviously not enough.

Iteration 5: Improving the "increased activity" weight

You can only have a whole amount of people tweeting in the past hour. Two or three. In our case, we needed something in the range of 2 1/2. The modified solution adds the number of posts divided by ten to the number of users, which means that currently, at least two people making at least three posts in an hour will determine a trending event in Zurich. This is not a perfect solution from the event discovery view, but it does what urgently needed to be done: prevent having too many tweets in the stream.

The next problem: we currently have four Twitter accounts that tweet events for these four cities. Our target was for each of them to make around 10 - 15 tweets a day, which seems like a number that is not spam. But how can a person see which of these events is THE event?

Iteration 6: Going super venue level 2

The latest version of the algorithm now recognizes two levels of events. An event (mostly 6 people on Foursquare, mostly 2 different people tweeting), and an outstanding event (around 12 people on Foursquare, around 4 people tweeting). Our goal was to make this super event happen only once a few days, on rare occasions two times per day, and it has already happened a few times.

Sometimes super events happen, with tens of posts in a single hour, such as the one for Philips Fashion week. These events definitely require more exposure.

The next iterations

At this point, I'm very satisfied with how the algorithm works, even though a few other modifications need to be done (specially to support different days of week specifics and behavior). By measuring what is happening, learning from that information and building the next releases based on that knowledge, the activity stream logic has come a long way from the initial version. Measuring is crucial, and rarely we have went to such extent to enable this in the widest way possible (e.g. the update to balancing the posts based on the previous events would be trivial by itself, but we wanted to log things that would happen but didn't happen, besides things that actually happened).

These cycles of Build - Measure - Learn can be a lot hard work, but they provide great results, which are also very fun and rewarding. Some people simply need to see how deep the rabbit hole is. Do you have any other interesting cases or experience with this approach?

Discover what's happening in Ljubljana in real-time

Fri, 28 Sep 2012 13:37:29 GMT

I've always been a big fan of the power of the crowds. When a mass of people can achieve much more than a few skilled individuals can. And ever since we've started playing with Twitter's API, I've been think about the possibilities of this magnificent data source. Besides Twenity, we've done a few other Twitter mashups like Twitter walls, but this wasn't enough. We wanted something more - geolocation. Displaying information on a map in real-time. But since there aren't that many tweets equipped with GPS coordinates, we needed to include other services for more diversity as well. Which we did, and Ljubljana Realtime, a social event discovery application, was born.

Launch

Data and services

Ljubljana Realtime currently feeds on four different services: Twitter, Foursquare, Instagram and Flickr. It would be great if we could add other services as well, but Facebook doesn't allow public geo search, Google+ doesn't support geo search at all, and other services either aren't appropriate or don't offer an API which would allow us to get their data.

These four services are visited once a minute, and all posts in a radius of around 5km from Ljubljana city center are found: tweets, Foursquare trending venues, pictures from Instagram and Flickr. A variety of information created with different purposes on different occasions.

The area covered by Ljubljana Realtime. Different services require different searches, based on maximum allowed radius.

The application

These posts are displayed on a map, which was the original idea for the prototype. The last hours of posts on Google Maps, which can be zoomed and filtered at will. But the whole display felt a bit chaotic (still does), since there are many posts in vicinity of one another. That's why we knew we need to group similar posts, and we did this by the post's nearest Foursquare venue. Then a funny thing happened: this simple solution enabled something magnificent, something that could be much bigger than the whole posts-on-a-map application.

Anyways, since geo location is heavily connected with mobile devices, the application is responsive and fully compatible with most smartphones. Perhaps there will be native apps as well at one point.

The stream

Back to the magnificent. Originally, a Twitter bot was intended to come with the application (I love making those), which would tweet all trending foursquare venues to promote the application. But this seemed a bit lame, we needed to add something cooler. Something that would add more value and detect an event before a 4sq trending venue would happen. This is where the mentioned grouping of posts by venue came in handy, and the logic is as follows: if two or more people publish form the same venue in a single hour, this could very well mean something's happening there. And in most occasions, this turned out to be true. Read further for more details.

Discovering an event before a trending venue on Foursquare.

The event discovery stream is available on Twitter and Facebook.

The problems

Besides unstable APIs, the biggest problem we are currently facing is the geolocation itself. GPS chips in mobile phones are often not accurate enough, so people are located tens or hundreds of meters from their actual location. Combine that with the amount of Foursquare venues out there (imagine tall buildings), and you can understand Ljubljana Realtime sometimes has problems with connecting a post to a venue. Not to mention duplicated venues. We've eliminated some of this effect by only using venues with a certain amount of checkins and different users, but this will surely be the greatest challenge the project is facing in the future.

An event which is not.

The results

Problems aside, in most cases, Ljubljana Realtime event discovery works great. In a week or so since it's online, it discovered many events that were happening in Ljubljana (night run to the Castle, an athletic meeting, one of the first iPhones 5 in Slovenia, a public garage sale in park Tabor, etc.), and on many occasions, it discovered these events before a trending venue happened on Foursquare. Which is great. The potential is obviously there, and newer, improved versions and algorithms will surely behave even better.

A few of the great discoveries Ljubljana Realtime made.

The plans

This project is being developed in an agile way, where the application's behavior is constantly being monitored and changes deployed rapidly according to discovered strengths and weaknesses. The MVP (minimum viable product) is there, and with a few minor modifications, Ljubljana Realtime will soon be ready to expand to other cities and regions. Now it's up to you to help us, and it's pretty simple. When something magical is happening on and you are tweeting about it anyways, be a sport and click the arrow to include your geolocation in the tweet. By doing this, you will help others to discover what's going on in our beautiful city.

That's it for now, party on.

p.s. for all the Slovenians out there: the coordinates embedded in a tweet are pretty accurate even though Twitter will say you are in Italy. If you look at the picture of the map below the tweet, there's a polygon around Italy which sadly contains Slovenia as well. Hopefully, Twitter will remove bug someday.

Finally, a reason for bloggers to use Google+

Sat, 22 Sep 2012 13:20:55 GMT

Despite the traffic, there isn't that much going on on Google+, and the referrals from this social network are still not that numerous. Most of mine come from other sources, but Google has a plan, and this plan is a smart one - using their services to push forward other services. You've probably noticed more and more results in Google search contain the author's picture. They stand out from the rest, and since most bloggers want to get as much traffic to their site as possible, this fact can make a difference between which link is clicked or not. If you ask me, setting this up is a must, and it's really easy to do. But you need to have and pimp your Google+ profile.

There are three steps you need to follow:

have an "Author" page, which links to your Google+ profile with rel="me"
have a link to your "Author" page on your post, which contains rel="author"
have a link from your Google+ profile to your "Author" page (in "Contributor to" section)

Here are the full instructions on how to technically implement this. It should take you about 10 minutes, and I've waited for about a week for Google's search results to actually start displaying my profile picture.

This feature has been available for quite some time, but I still see many bloggers that haven't implemented it yet, also because its availability was mostly unnoticed. Combining search and social is definitely an advantage Google (still) has over Facebook, and they should've made more promotion about this interesting upgrade. Bloggers are content creators, and you want as many content creators inside your social network as possible. They are the ones who provide original thoughts that others are looking for and will eventually follow.

It's hard to estimate if this fact (and different pictures) will have any significant impact on search traffic, I'll keep you posted. But before everyone has it, it's a nice little feature that makes your results look different and much more appealing. Good job, Google, content is king, and you should stay its steward.

The art of internal hyperlinking

Sun, 04 Sep 2011 21:00:02 GMT

Everybody wants traffic on their blog or website. Direct traffic. Referring traffic. Search traffic. All good in their own way. Direct traffic means having a strong brand. Referring traffic means having a strong network. Search traffic means having a strong team. The first two are hard to influence, but search - that's the one you can influence the most. And even if search technology has changed a lot in the past few years, with mathematical algorithms slowly getting replaced by social ones, old school search engine optimization can still make a difference.

Let's say you already have good content. And you have it technically optimized and search engine friendly. Like, Tweet and +1 buttons implemented where applicable, since they are more and more important for search result rankings. All good, but you might have missed something. Since it's very hard to persuade other people to link to your site giving you a higher PageRank, you just have to do it yourself. I'm talking about internal hyperlinking, where you cross-reference the content you already have. Google loves it so much that their official blog, besides influential blogs such as Mashable and ReadWriteWeb, have main headings referencing itself.

I approached this issue from two different angles. Sometime in October 2010 I took the time to systematically cross reference all my blog posts. It took me a lot of time, but I think it was worth it. Besides, I developed a "similar content" module, which additionally does it for me. The results on the diagram below, which display search-based traffic on my blog, are not real proof of that fact, since the experiment was not fully scientific; there are many other SEO factors, additional content and optimizations that were put into the equation. But still, most energy was put into content cross referencing and the trend looks pretty good.

Traffic to http://stritar.net via search engines

Since internal hyperlinking is an ongoing process, you have to have a good overview about the content you own, something that provides you with the complete picture. I used a plain Excel file, grouping my posts into general categories and started drawing arrows representing hyperlinks. At this point, it's already become a bit of a mess, so I'm thinking about moving to a stronger diagram-oriented software, but I think you get the picture.

http://stritar.net internal hyperlink structure

Internal hyperlinking can help, and it's something you can do even if you don't have a lot of technical skills. Just a lot of time, but we're in a recession anyways. Now go for it and let me know about the results.

Please help me upgrade my Twitter bot

Tue, 30 Aug 2011 20:33:12 GMT

Half a year ago I decided to make something out of my Delicious bookmarks. The magazine-style display inspired by Flipboard wasn't enough, I wanted to publish these links somewhere outside my chronolog, somewhere on Twitter. So I made a bot. It's doing quite well, posting like mad, but it's really not where I want it to be. Until now, it made about 3.000 tweets (around 500 per month), but has only 67 followers. I know my taste in content is a bit obscure, but still, only 67 followers?

This calls for an upgrade. And since I won't change my interests and bookmarking habits, something else needs to be done. That's where I need your advice. Crowdsourcing the concept and stuff.

The brand

This is probably one of the greatest fails of the project. At first, I thought of it as an extension of my blog. Hence the account @stritar_net, the name Stritar's chronolog, together with the description it has. Should I rename it and try to make it a standalone "brand"? Should I openly say it's a bot (Stritar's bot or something)?

The selection

The bot currently posts ALL my bookmarks to Twitter (as mentioned, around 15 per day or 500 per month) without any selection. But it could be done. Since my bookmarks are originally tagged, it could leave out those with too few tags (since I use the same method of counting tags to determine the initial weight of content for the magazine). Or specialize in specific segments according to tags. There could even be more of them bots. What do you think?

The frequency

Currently, the bookmarks I make go into a queue. I hate those Twitter accounts that post 10 tweets in 5 minutes and go silent for a day. I wanted to make it more smooth. So the queue always knows how many items it holds and adapts the frequency of posting according to it (less bookmarks in queue mean less frequent tweets). But that produces the situation where most of them are already a few hours or days out of date when they are published. A higher publishing frequency would solve some of it, but it opens a great dilemma: what's the lesser evil, over-spamming or being out of date?

The order

The order of bookmarks posted on Twitter is determined by two factors. The number of tags and the date they were published. More tags equals more importance. Older bookmarks get published sooner, otherwise they would get even more out of date. Should I do it the other way around and post more recent links sooner? This would make some of them more interesting and up-to-date, but other worse. Breaking content or consolidated content?

So many decisions… In this case, the best way probably doesn't exists, but trade-offs can always be decided for the better. Please let me know what you think, your help would be more than appreciated. I could help back if I can.

Determining if an element is in the last row of a table

Mon, 25 Jul 2011 11:57:52 GMT

Once upon a time I stumbled upon a problem, where I needed to calculate if an element is in the last row of a table. Here's the scenario: you have a number of items, which are put in a table from left to right. When the row is full, the items continue in the next row. Imagine an airplane or a theater where people start sitting front-left and continue to the right until they run out of space, then going to the next row and so on. Now we want to know which people are sitting in the last of the populated rows.

A weird problem, but hopefully I will be able to show you some cool results produced by this algorithm someday (yes, it's usable).

The equation has 2 parameters: the total number of elements and the number of columns in a single row. There are a few ways to do it, using division with remainders (the modulo operation). The simple way would be comparing the total number of rows with the element's current row. Another one would be to calculate the number of elements in the last row and see if our element is in those last few. While both seem logically easy, they actually suck, because they contain exceptions (some states need to be handled specifically - if the last row is full or not).

That's why I went for the ultimate way, using abstract mathematics which requires real magic, and since it would be too boring to explain it, just take it if you need it and don't try to understand - to be honest, even I'm not perfectly sure how I did it.

The number of rows way:

lastRow = ((Math.DivRem(currentElementIndex + 1, numberOfColumns, out remainder1) + Convert.ToInt16(remainder1 > 0)) >= Math.DivRem(totalElements, numberOfColumns, out remainder2) + Convert.ToInt16(remainder2 > 0))
* Math.DivRem returns the division result and the remainder, since both are required for the calculation

The number of items in the last row way:

lastRow = (totalElements - (totalElements % numberOfColumns) - (numberOfColumns * Convert.ToInt16((totalElements % numberOfColumns) == 0)) < currentElementIndex + 1)
* % is the modulo - remainder from dividing

The ultimate way:

lastRow = totalElements < ((numberOfColumns + currentElementIndex) - (currentElementIndex % numberOfColumns) + 1)
* % is the modulo - remainder from dividing

Mathematics is awesome.

UPDATE (27.7.2011): Silly me. Overwhelmed by "The ultimate way", I missed the opportunity to simplify "The number of rows way". I guess this is one of the cases which explain why counting starts with 0 instead of 1 in programming.

The improved number of rows way:

lastRow = Math.DivRem(currentElementIndex, numberOfColumns, out result1) >= Math.DivRem(totalElements - 1, numberOfColumns, out result2))
* Math.DivRem returns the division result and the remainder, since we need the number rounded down.

Adding share buttons to your blog or website - a comprehensive guide

Mon, 06 Jun 2011 12:05:14 GMT

A lot has happened in the field of share buttons in the past year: Google +1 button for web pages was introduced, Facebook started to migrate the Share and Like buttons, TweetMeme button is slowly getting replaced by the offical Tweet button, etc. Enough to make maintenance of these buttons a pain in the ass. But since social activity is getting more and more important for SEO, this needs to be done, one way or another. To make it easier, I've put together a comprehensive list of different share widgets, together with some explanation, sample code and direct links to full documentation.

Keep it simple: Use basic links

Most social services support direct linking to share forms, which can be populated using a proper request. This means the URL of the target content (and sometimes title) must be passed in the query string (e.g. "?url=http://stritar.net"). In case you require a simple solution that doesn't require a lot of space, or you would like to style your share buttons on your own, this could be what you're looking for.

Facebook: http://facebook.com/sharer.php?u=<%=contentUrl%>
Twitter: http://twitter.com/?status=<%=contentTitle%> <%=contentUrl%>
Reddit: http://reddit.com/submit?url=<%=contentUrl%>&title=<%=contentTitle%>
Digg: http://digg.com/submit?url=<%=contentUrl%>
Delicious: http://delicious.com/post?url=<%=contentUrl%>&title=<%=contentTitle%>
StumbleUpon: http://stumbleupon.com/submit?url=<%=contentUrl%>&title=<%=contentTitle%>

Keep it simpler: Use AddThis

If you're not a demanding users and you're OK with a plain solution, the service AddThis may be just what you need. Set the parameters and you're good to go, and the number of supported services is really huge (and you also get the Print button).

Result:

Possible modifications: too many to mention

More info: http://www.addthis.com/

Facebook

So, you'll rather go for the real thing? Let's begin with Facebook. If you're trying to get the code for the Like button and it says you have to be a registered developer, don't worry, just logout and everything will be great. After that you will be offered with two sets of code, both of them work. I used the second one, which also support "Send to".

<div id="fb-root"></div>
<script src="http://connect.facebook.net/en_US/all.js#xfbml=1"></script>
<fb:like href="<%=contentUrl%>" send="false" layout="button_count" width="130" show_faces="false" action="like" font=""></fb:like>

Result:

Possible modifications: layout (standard, button_count, box_count), colorscheme (light, dark), action (like, recommend), show_faces, font, width, send (add send button), etc.

More info: http://developers.facebook.com/docs/reference/plugins/like/

Twitter

Since Twitter introduced the new Tweet button, the original TweetMeme button started to behave strangely (sometimes it doesn't count tweets correctly). On the other hand, the official Tweet button doesn't count the tweets for older posts, so all your viral posts from the past will show 0 tweets (and I was so proud one of my posts got more than 100 retweets!). Your choice, I've decided to go for the new one.

<a href="http://twitter.com/share" class="twitter-share-button" data-url="<%=contentUrl%>" data-count="horizontal">Tweet</a>
<script type="text/javascript" src="http://platform.twitter.com/widgets.js"></script>

Result:

TweetMeme:

Possible modifications: data-count (vertical, horizontal, none), which modifies the layout. You can also set the default text and specify @via, etc.

More info: http://twitter.com/goodies/tweetbutton

Reddit is a great community that can get you quite a bit of traffic if your topic is more on the geeky side.

Result:

Possible modifications: tens of different layouts

More info: http://www.reddit.com/buttons/

Digg

Since Digg's last upgrade, I haven't been seeing any traffic from it, but it's nice to dream about the old times. Not working properly on FF2.

Result:

Possible modifications: class (DiggWide, DiggMedium, DiggCompact, DiggIcon), which modifies the layout.

More info: http://about.digg.com/downloads/button/smart

Delicious

We'll probably be seing a new version of Delicious button soon, but for now, you can either put a simple link without the count to your blog or some hacking is required. You practically have to make your own button that retrieves the data from Delicious and puts it into correct HTML tags. For advanced users only.

<span id="deliciouscount" style="font-size: 12px; font-weight: bold;"></span>
<a id="deliciouslink" style="font-size: 12px; font-weight: bold; text-decoration: none;" target="_blank" title="View details"></a>
<script type='text/javascript'>
function displayURL(data) {
var urlinfo = data[0];
if (urlinfo == null) {
document.getElementById('deliciouscount').innerHTML = "Bookmarks: 0";
return;
}
else {
document.getElementById('deliciouslink').innerHTML = "Bookmarks: " + urlinfo.total_posts;
document.getElementById('deliciouslink').href = 'http://delicious.com/url/' + urlinfo.hash;
}
}
</script>
<script src="http://badges.del.icio.us/feeds/json/url/data?url=<%=contentUrl%>&callback=displayURL" ></script>

Result:

Possible modifications: the world is not enough

More info: contact me

StumbleUpon

I'm not sure if I'm using StumbleUpon properly, but I haven't managed to get a single stumble since I started blogging, so I temporarily removed it from my blog (due to lack of space).

Result:

Possible modifications: many different layouts

More info: http://www.stumbleupon.com/badges/

Google +1

It's too soon to say if +1 button will be a game changer, or it's just too lame, too late, like other Google's social services. We'll see. Not working properly on FF2, FF3.5 and IE7.

Result:

Possible modifications: size (small, standard, medium, tall)

More info: http://www.google.com/webmasters/+1/button/index.html

LinkedIn (Update 28.11.2011)

LinkedIn may not be your dynamic social platform, but perhaps you may still find use for it.

Result:

Possible modifications: layour (vertical, horizontal, nocount)

More info: https://developer.linkedin.com/plugins/share-button

Special: Like Facebook page and follow on Twitter

If you have a Facebook page, all you need to do is point the Like button to the URL of the page, and people will automatically become "fans". Similarly, you can implement the new Twitter Follow button and hopefully get new followers.

<a href="http://twitter.com/gstritar" class="twitter-follow-button" data-show-count="false" >Follow @gstritar</a>
<script src="http://platform.twitter.com/widgets.js" type="text/javascript"></script>

Follow @gstritar

Other platforms

All of these buttons can be modified to some extent, and most of them will work without the specified URL. But you'll probably need to set it anyways so they will also work on your homepage with many posts.

The cases presented were made for asp.net, but they can be modified for other platforms such as WordPress or Blogger by replacing the <%=contentUrl%> with something like <?php the_permalink() ?> or data:post.url (sorry, no experience). But if you managed to get the Facebook Like button to work, you will surely be able to modify others too.

There you go, time to share.

Reinventing SEO: The social media effect

Tue, 19 Apr 2011 09:00:38 GMT

Looks like the time for classic search and Search Engine Optimization (SEO) is slowly running out, waiting to be replaced by more advanced and efficient algorithms than mathematical - human powered. The Facebook Like button, the Twitter retweet button and other social share widgets are on the uprise, and Google is fighting back with all its might. For now, their business model relies heavily on search (other project like Android and Chrome too), but their future social success was named the number one priority by the new old CEO Larry Page.

Google still has a problem with providing a solid social alternative to the newly crowned social players, and for now (we'll have to see what happens with the new +1 button), all they can do is to somehow play along. In the mean time, a lot of people have been noticing the impact of Facebook likes and Tweets on their Google results ranking, and guess what – I've noticed the same thing. Welcome to the age of social curation, where rating content is slowly getting as important as generating content. I guess there's about a billion times too much of it online, and who else knows it better than Google and Bing.

While we knew Google is very good at adapting its services to new trends, we are quite happy they actually went this far, embracing social virality into their search results. But what happened to the world's most powerful mathematic algorithm, Google's PageRank? Is it becoming obsolete to the Facebook Like's search algorithm, which will surely come around soon in its full glory? Actually, it's getting clear it became obsolete together with the static Web 1.0, but only to get reborn for the social world of Web 2.0. Slowly, almost underground, while we were being fed with news about how loading speeds make a significant difference on Google ranking.

I've written a few blog posts that have been generating some social buzz. A few Tweets, Likes and Reddit upvotes, and you have a winner (thank you!). "Jeopardy Slovenia" may not be a power search, but it became the first result on Google the same day I've published the post. With PageRank 0! And "Apple Slovenia", a keyword much more interesting, is also displaying my post on the first page. Not bad. And that's something we will probably be seeing even more of in the future, and that's why blogging is still (if not more than ever) very important.

It's time to start monetizing your social capital. And of course, upgrade SEO marketing strategies with more efficient SMO (Social Media Optimization) strategies. Before you'll get your page on top of Google the old school way, you'll grow old. So be cool and remember, sharing is caring. Yes, that means you should click the button.

I've developed a magazine based on my Delicious bookmarks. And a Twitter bot.

Fri, 25 Feb 2011 07:34:38 GMT

Last week was marked a great social achievement of mine – I managed to bookmark my 10.000th bookmark on Delicious. A lot of people have 10.000 tweets, but not many own 10.000 bookmarks, fully tagged and classified. I've been collecting these since December 2006 (probably one od my first 2.0 addictions), and they are becoming one of my greatest possessions – knowledge is the ultimate collection. Hopefully Delicious won't get shut down or left behind, so I will be able to continue with this obsession.

I've made a few interesting mashups with them already; as my most frequent actions online, my bookmarks represent the core of the chronolog. But things are changing fast and the desire for presenting information is moving into a new dimension. In case you didn't notice, tablets are mainstream, and the media industry already hopes they are the solution they've been waiting for. The Daily, which was released a few weeks ago, is the first no-print, tablet only magazine available (for iPad, Android version is coming), and other applications, based on social curation, such as Flipboard (which is amazing and inspired all of this) and OnSwipe are revolutionizing the way we (create and) consume content.

I find this evolution of displaying information fascinating and a bit ironic - going from newspaper form to blog form and back - but I've decided to play along, developing a magazine based on my bookmarks. Using the Html Agility Pack library for asp.net I managed to extract an image and a few paragraphs from each URL I bookmarked, using them to form a magazine. The number of tags I put on each link determines the initial weight, which is additionally modified with your views and likes, allowing it to constantly adapt its shape. Those links which are stronger, are displayed higher, have a bigger picture and more text which makes them more visible.

While contemplating about the potential of different APIs and all the pages I bookmarked these years, it occurred to me I could also make a Twitter bot. All the bookmarks I save are now getting automatically shortened with bit.ly and posted to Twitter, with a bit of artificial intelligence. The speed of posting is determined by the number of items waiting in the queue, adjusting to the frequency of my actions. It still has a few problems, but they are only appropriate for a prototype, whose posting algorithm still needs to be technically and mathematically improved.

The life cycle of my links I like has become quite a ride, as you can see in the diagram below, similar as the evolution of the Web, transformed by social, mobile and the upcoming domination of the tablets. And while the major players are able to spend millions on the development of new and creative solutions, small players such as myself can only play along. Luckily, I like to play.

The path each one of my bookmarks makes in its lifetime.

The chronolog now understands connections between content

Wed, 03 Nov 2010 21:20:56 GMT

I once made a promise that I will try to incorporate as many interesting features as possible into my blog. My previous development sessions were based mostly on interactions of readers with the posts, the peak of it being the Hot on the chronolog algorithm. But now, as the chronolog finally reached critical mass in the amount of content it operates with, the time has come to do something new. The next step is focused on a different functionality, and a few days ago, the chronolog received an algorithm for recognizing relationships between different blog posts.

The connections

The whole concept is based on the occurrences of categories (which are actually tags) on different blog posts, the most obvious being the number of the same tags two different posts share. We did something similar on a web portal we launched a few months ago, and it works pretty well. Sure, the proper way to do it would be using real text mining, where the strength of the relationships would be based on meaning and occurrences of words and external hyperlinks in a specific post. But in this stage, I'm keeping it simple: if two posts share a lot of tags, they appear more related.

The weight

Since some categories (tags) are used more often, they appear in many posts, making these posts too heavily related with each other. The number of categories attached to a single post also varies, giving a post with many tags a much stronger chance to appear as related to another. Therefore the general equation contains two modifiers, which are giving weight to each shared tag between two posts.

Categories that appear only a few times globally, have more weight, because they represent a more scarce and therefore a more interesting and stronger connection. This takes care of the tags which are used very often, making them not too dominant. On the other hand, the weight of each tag on a post drops with the total number of tags the post has, so those posts, which have a lot of tags, don't become every other's related post. It may sound confusing, but it's probably a bit simpler to develop than to explain.

The results

I was actually quite surprised about the result the algorithm makes (which you can now see on the bottom of every post). As I was playing around a bit, observing how the calculation behaves and playing with constants, I actually found some interesting connections between posts which I didn't notice before. The engine finds quite a strong relationship between the post about using Web 2.0 logos in TV commercials and the one about Round browser icons, both of them being design clichés. The case of Microsoft and Google going social also made it strong, as the two posts are describing the struggle of two technology giants trying to adapt to the new situation. I could go on and on, but than you would probably just say I was doing SEO too hard.

Search Engine Optimization (SEO) is actually another hidden benefit of the feature, something that occurred to me after I've already finished working on it. Google likes it if you have your content internally cross linked, so what better way to do it than to have automation take care of it. So until SEO dies, this new functionality is actually a double win, because the chronolog became more optimized for crawlers and hopefully more useful for the readers. Even though most of you probably won't even notice.

The awesomeness of the Facebook Like button

Mon, 28 Jun 2010 07:46:07 GMT

Usually I'm quite critical towards Facebook and their policy for dominating the World Wide Web, but in this case I have to give them credit and respect. I finally took the time to add advanced share buttons / badges to my blog (and actually made a whole science out of it, but more on that some other time), one of them being the infamous Facebook Like button. Needless to say it's probably the most advanced available share widget from the technical point of view, but I never imagined it's that advanced. Looks like Facebook isn't loosing any time to index the web, and their architecture for achieving this goal is set up very well. The Like button is fully connected with all other social activities on Facebook, which obviously means Facebook is building a giant man powered map of the World Wide Web.

After the Facebook Like button was released 21.4.2010 on the f8 conference as a part of the new Open Graph protocol, it was quickly adopted by a lot of sites and portals, many of them mainstream. It didn't take long for the button to produce criticism, because it turned out it can be used to exploit user profiles by installing applications and gathering user data. Today, this debacle is slowly turning into history, and more than 100,001 sites already use the Facebook Like button, producing probably unimaginable amounts of likes and driving massive traffic to their sites.

I use Facebook (and other social sites) to promote my blog by sharing links, and actually get quite a few referrers from it. Many of my previous posts have been liked and commented, making it a very useful tool for generating visits. But I haven't imagined these interactions would also reflect on the Facebook Like button I finally adopted on my blog. I apologize if this is a well-known fact, but I haven't heard about it from anyone and still have trouble finding any mention of it.

When I first tested how the Facebook Like button behaves, I noticed it displayed a few likes even though I've clicked on it around 2 AM. This made me wonder if something is wrong with it. Is Facebook trying to fake likes to make the button more appealing for publishers? Surely they have to be smarter than that. After I dug in a little deeper I noticed it displays a different number of likes on different posts I've made and published to Facebook. Hm... Could this mean...?

I was intrigued by what I found out. Besides the actual clicks, the Like button also displays all the likes and comments on the links you or anybody else shared on Facebook, making it a global Facebook social interactions counter with a specific URL (link). I manually went through the statistics and concluded that all shares, likes and comments of posted links are also converted into likes on the button. I have tested it with 10 different posts and the numbers are pretty much correct, and I even found some proof of this fact. The people behind these likes and comments aren't mentioned with their names on the Like button, but they are there in the number. A pretty awesome feature for making your content look cooler and getting the most out of social interactions on Facebook.

Here are some results from my testing, which provide almost a perfect match between likes on the freshly published (and unclicked) Like button and shares, likes and comments inside Facebook.

The influence of new generation information systems on modern organizations: 5 likes on the button vs. 1 share (me), 2 likes (A, B) and 2 comments (me, B) - 5:5
Facebook vs. Twitter - Part 2: Privacy and real-time web: 6 likes on the button vs. 1 share (me) and 5 likes (A, B, C, D, E) - 6:6
Why are all browser logos and icons round (and blue)?: 8 likes on the button vs. 3 shares (me, A, unknown), 4 likes (A, B, C, D) on my share, 1 like on A's share - 8:8
Arthur C. Clarke envisioning the World Wide Web in 1968: 19 likes on the button vs. 8 shares (me, 7 unknown), 4 likes (A, B, C, D) and 2 comments (C, me) on my share, other interactions unknown - 19: 14 + x
Neolab in 2009: 19 likes on the button vs. 1 share (me), 9 likes (A - I), 8 comments (A, 2 x me, J - M) - 19:18

Even though there are some differences, I still think the explanation is more than satisfactory and if I'm right, this fact alone probably makes it worth implementing anywhere. Besides, the Like button supposedly produces more traffic on sites and blogs using it, which will definitely be an interesting statistic to monitor. The only sad thing about is that I had to like (and unlike) all my previous blog posts to activate the count (taking some time and producing an empty Facebook profile), and it turns out sometimes the numbers don't load correctly because of unknown reasons.

My fascination with the little widget doesn't mean Facebook is off the hook from my side. As useful as the Like button may be for companies, publishers and bloggers, it probably isn't that good for the users and their privacy. Facebook should build a fair and transparent strategy covering these issues, because it becoming Big Brother surely isn't that promising for the people and the future of the World Wide Web, and more and more of us are aware of that. But taking a look from the other perspective and comparing it to similar sharing solutions, the Facebook Like button is pure awesomeness.

UPDATE (27.5.2011): Ever since Facebook started with the unification of the Like and Share buttons, this behavior seems to have vanished. But it was fun while it lasted.

UPDATE (1.11.2011): Silly me. The behavior is still there, but I've started posting to Facebook using an url shortener, which messes up the global like count. I guess it's a trade-off between better measurement and more likes displayed on your post.

The impact of hyperlinks, toolbars and URL shorteners on Google Analytics

Tue, 09 Mar 2010 18:51:45 GMT

In the past few weeks I've done an extended analysis of visits on my blog, which made me wonder how the super fancy new web gadgets and features influence Google Analytics and traffic reports. By these new gadgets I mean the nowadays very popular URL shorteners, such as tinyurl or bit.ly, and the annoying inside-browser toolbars, used by Digg, Stumbleupon, Google images and other services. These inventions made me wonder, as well as probably many other bloggers, web developers and marketers do - are these things messing up the traffic statistics? To be sure, I had to try it out by myself and found out the following: No, they do not. Or better put, Google is smart enough to know what's happening.

The methodology

Google Analytics and other statistical software is based on combining the user agent (operating system, browser), IP and browser cookies to calculate visits on a site. While others are captured for different information, cookies are still the base of elementary web metrics, so it's crucial to understand them and have them under control. I did just that each time I did a test – clean Google's cookies, close the browser, open the browser, check it out. I made a hidden post for testing, one which definitely wouldn't get any other referrers other than me, besides using some old and already forgotten posts I made on Digg months ago and some indexed by Google images. Then I started testing different cases, which would help me understand the behavior of all the above mentioned things.

Referring vs. navigating

The first interesting thing I found out was that Google Analytics knows the difference between clicking on a link and manually navigating to a page by entering the URL in the browser. In the first case it recognizes the referrer, and in other one it doesn't (it is displayed as google / organic in the statistics). This is caused by the referrer information captured in the HTML header of every web page. So, if somebody manually enters your web page's address after seeing a link on Facebook, Facebook won't be counted as a referrer, but if they click on the link, it will be.

Url shorteners

I tested a few url shorteners, tinyurl, is.gd, skrci.me and the one Twitter automatically uses (bit.ly). I found out that the ones I manually created and clicked on them inside the URL shortening site showed this site as the referrer. But in the case of Twitter, on which I made a tweet, clicked on the link and deleted it within seconds, Twitter was correctly shown as the referrer, even though the click first went to bit.ly and than to my blog. I went further, created a new shortened URL, put it in a hyperlink on a server, clicked on it, and again, this server was shown as the referrer. Because url shorteners only make the redirect, the click is keeping the original referrer, which enables the referrers to be fully captured, even if they go through the shortened URL.

Toolbars

I use a few Web 2.0 services to promote my blog, and it's been a trend for them to provide toolbars, which display the target site inside the parent site. The main reason for them doing this is to keep users inside their site, and in my opinion, it's annoying and it sucks. But at least it doesn't influence the statistics. I tested this behavior on Digg toolbar and on Google images (without removing or closing the toolbar) and in both cases it worked perfectly – the referrer was correctly recognized. After all, upon technically examining both cases, it's only an iframe opening the designated page below the toolbar, so the target page actually does fully open anyways.

Proof

The screenshots of Google Analytics below support my theory. In the case of the hidden post, I managed to create the following situations which prove my discussed behaviour of hyperlinks and URL shorteners:

I clicked on a link from neolab.si (one referrer from neolab.si),
I manually navigated to the post from neolab.si twice (by entering the url in the browser while being on neolab.si), shown as google / organic,
I clicked on a shortened link on the is.gd site, later I pasted another is.gd short url directly to my browser (two referrers from is.gd),
I clicked on a shortened link on the tinyurl site (a referrer from tinyurl),
I clicked on the automatically bit.ly generated url in a tweet (a referrer from Twitter),
I generated the skrci.me short url, put it in a hyperlink on localhost and clicked on it (a referrer from localhost).

The second report displays the referrers from Digg and Google images, both services using toolbars. On the first occasion, I opened two different posts inside the Digg toolbar, and on the second, two posts inside Google images toolbar, all of them without closing the toolbar (I went for one post twice, to check out if a session is also created and found out it is). As you can see, the referrers are all there and the toolbars don't corrupt the data in any way.

Conclusion

Look like we don't need to worry about these things anymore. I'm actually quite surprised about finding out the mentioned things work like a charm, not influencing the analysis and statistics in any case. From now on, there can be no more blaming these new features and gadgets on low traffic and weird referrers. The World Wide Web has been well planned and Google Analytics is able to know everything, so if your statistics seem weird, there is probably more chance that you are the one who's wrong.