Analytics Expert Blog

The new Clicky is live! In the form of a novel, here’s what new:

• Cookies

  We now use cookies to more accurately track unique visitors. As mentioned previously, you can disable cookies if you don’t want them on your site, with clicky_custom.no_cookies. If a visitor does not have cookies enabled or you have disabled them on your site, then we fallback on the visitor’s IP address, which is how we were doing this previously.

• Pinging

  When a visitor remains on a single page of your site, our tracking code will ping our tracking servers in the background so we know they are still online. This will also let us give you much more accurate time online values, both per visitor and average.

  By default, a visitor will ping us for 20 minutes while on one page. Depending on the type of site you are running, you may wish for a longer time that that. Don’t worry, you can customize it! Using clicky_custom.timeout, you can extend the pinging time up to 4 hours. This is perfect for sites that are focused on things like videos or games.

• Spy + Ping

  Spy uses the new pinging functionality to more accurately display who is actually on your site right now. Previously, we were just using a 5 minute timeout period of no activity to determine that a visitor was gone. Pinging allows us to actually know who is on your site, so Spy will be much more accurate. Pinging starts at every 15 seconds but quickly decays to once per minute, so there’s still a timeout of 1 minute on Spy before a visitor will disappear. Pinging will also let us display visitors who are just sitting there on one page. Previosly they would disappear after 5 minutes but now they will stay there as long as they keep pinging us.

  The result of this is a lot more spikes in the visitors online graph, rather than a fairly smooth one as you may be used to. Here’s an example of what we mean:

  

• New vs returning visitors

  We now track new vs returning visitors. You will see in the basics dashboard module, there’s a new expand link. Clicking this will show your unique visitors and your new visitors for the date or date range you are viewing. Because we are only tracking new visitors from this point forward, this metric won’t be terribly accurate for the first week or two until most of your regulars have visited your site again.

  You can also filter by new or returning visitors, it’s in the filter drop down box on the visitors page.

  

• Bounce rates

  Our bounce rate calculations also take pinging into account now. As far as we know, all other analytics services define a bounce as a visitor who only views one page. But if they actually stay on your site for a little bit, they are probably more engaged with your site. The first ping occurs after 15 seconds, so that’s our new threshold for determining what a bounce is. Anyone who is on your site less than 15 seconds is a bounce, anyone on longer is NOT a bounce – regardless of how many actions they had. We think this is a much better way to calculate this metric.

• Self hosted tracking code

  If you are self-hosting the tracking code, you will need to grab a fresh copy to get cookie and ping support. We’d also like to make clear that we no longer support this option. Our tracking code is now on a CDN and we also offer asynchronous tracking code, so there’s no need for it anymore. Making this update backwards compatible for people doing the self-hosted thing was a bit of a pain. From this point on, we cannot guarantee backwards compatibility with those of your self-hosting, so you are doing so at your own risk!

• More efficient

  The way we process and store visitors has dramatically changed, and the result is about an 80% increase in efficiency – which is a very big deal. This means we’ll have to do database maintenance a lot less often (e.g. what we do every few months to a couple of servers, such as last week with db7/8/9). Smaller storage also means much faster filtering. You should notice a dramatic increase in filtering (segmentation) of your visitors for most data types.

  This also results in much faster processing of traffic. We think it’s going to allow us to track higher traffic sites. The highest level of traffic we allow for any site is 500,000 page views per day. We think our new system will be able to double that capacity – but we’re not sure yet. We’re going to be monitoring things and will certainly let people know if we increase this limit.

  Please note, converting your existing visitors to this new format will take up to 48 hours on some servers. So rather than keep these servers offline during the conversion, the conversion will be occuring over the weekend while the servers are live. It should be finished by Sunday though. The point is, when viewing your visitors list, some visitors may have incomplete data (e.g. just an IP address). Don’t panic – their data *will* appear! We’re doing this from newest to oldest visitors, so you probably won’t even notice it unless you go back in your history a bit.

There may be a few bugs lying around. We’ll be monitoring and fixing anything unusual over the weekend. If you see anything strange, don’t hesitate to leave a comment here or send us an email. Hope you enjoy!

The newest version of Clicky is being released tonight. There are some pretty big database changes so we’ll be taking all database servers offline for approximately two hours while the changes are made. During this time, stats will be unavailable, but incoming traffic data will of course still be logged, and will be processed once the servers come back online.

This process will start tonight (Friday) around 8 or 9 PM PST (GMT -8). We’ll be posting a full run down of all the new features once it’s been released.

Follow us on Twitter for real time updates during the process.

We need to perform maintenance on a few database servers this weekend. This will begin on Friday night around 8-10 PM (USA PST), and will last 8-10 hours. Affected servers are db7, 8, and 9. The database server that any site is hosted on can be found on that site’s main preference page.

These are all fairly old servers, although they were opened up for some new sites for a couple of weeks in the beginning of 2010. Usually we only have to do this kind of maintenance on older servers so it doesn’t affect too many people (since typically speaking, the longer you’ve been a member, the less often you log on). Unfortunately, this is not the case for everyone this time, since these servers have some sites registered as recently as 3 months ago.

When we do this type of maintenance we usually take the affected servers fully offline. However, due to complaints we are going to try a new method this time where they stay online so you can still view your stats, but no new data will be processed while the maintenance is happening. This will slow down the maintenance to some degree, but we think it will still be acceptably fast, especially since Friday night / Saturday morning is the lowest traffic levels of the week. We will be monitoring the performance and if we decide it’s slowing it down too much, we will take them offline so that they can finish as quickly as possible, but we hope to not have to do that.

Our new release, that will include pinging and cookie support in the tracking code, much more accurate unique visitor and time online values, as well as tracking new vs returning visitors, is coming along really well. We were hoping to get it out the door this weekend, but combined with the database maintenance, it is just too much. So we are delaying it until next weekend. There are a couple of major database changes that in this new release, so every database server will have to be taken fully offline for approximately 2 hours while the changes are made. We plan to do this on a Friday night as well, so the least number of people are affected.

During maintenance periods, your traffic data is still logged, but it is not processed until the database comes back online. So don’t worry, you won’t lose any stats! But they will be lagging behind real time when they come back online for at least a couple of hours.

We are also always tweeting live updates during maintenance, so be sure to follow us on Twitter to for up the minute updates.

You thought the sneak peek of our new interface was hot? You ain’t seen nothin’ yet!

You like Foursquare. You like Clicky. Why not combine the two? Exactly what we were thinking! Every time you visit a site with the Clicky tracking code installed, we’ll automatically tweet out from your Twitter account which web site you are viewing. Combined with Foursquare, not only will all your followers know WHERE you are every second of every day, but also what you’re doing on the internet – every second of every day!

And what would be a check in service without super amazing badges to earn? Here’s a preview!

Old habits die hard

Awarded for logging on to any web site with IE6

OCD

Awarded for logging on to your own web site 20+ times in one day

Going to store ROFL

Awarded for logging on to Twitter 50+ times in one day and sharing way too much personal information with your followers

I can has cheezburger?

Awarded for uploading 20+ photos of your cat to any photo sharing web site

Reality Distortion Field

Awarded for purchasing an 10+ iPads on apple.com within 2 weeks of launch

Night owl

Awarded for using the internet non-stop between 12am and 5am

Lonely

Awarded for logging on to 50+ naughty web sites sites in one week

Lookin’ for Love

Awarded for visiting any craigslist personals category

Pizzaholic

Awarded for ordering pizza online 7 nights in a row

Swarm

Awarded for being part of the swarm from slashdot/digg/reddit that brings down a web site

That’s only a small preview. We’ll have over 5,000 badges ready to go when this feature launches next week. Hope you love it!

Competition in our space is heating up, so we’re working furiously to make Clicky stand out. One way to do that is with colors. Remember Paint by Numbers? Well we have taken that concept and color-coded all your favorite stats to the colors of the rainbow. And we thought… what’s prettier than a rainbow? Maybe two rainbows? Exactly.

So as you can see, your stats have never been prettier, or easier to read. Our guess is that you will never want to leave your computer again.

To say that we are

SUPER EXCITED

would be the understatement of the decade. This bad boy is going to eat our competition for breakfast. Coming soon!!!

Much requested, way too long coming, you can finally print out a receipt for any previous payment you have made to Clicky. To access this, click the View payments link in the top right corner of your user homepage, and then click the print receipt link next to any payment at the bottom of the page.

You’ve always received receipts via email for any payment made. But for many of you, this is not sufficient, particular in Europe where you need your VAT# on pretty much everything. With this new receipt on demand method, there is a single large form field for you to fill out all of the billing information that you need displayed on the receipt. Then click print, boom, you’re done.

We want to make Clicky have as little impact on your site as possible. We made some changes to our tracking code and tracking servers recently that have dramatically sped up the experience on your site. However, there’s one last thing that could still slow down your site a bit, and that’s the fact that we’re not using a CDN (Content Delivery Network) to distribute our tracking code.

The advantages of a CDN are clear – our static files would be hosted on servers all around the world so whenever someone visits your site, they pull the code from a server that’s very close to them.

But the disadvantages of a CDN are a real drag. First, none of them support SSL with your own domain name. If you want SSL, you have to use their domain. I understand why this is the case, but it’s unacceptable to me. Second, none of them support dynamic content (e.g., a PHP script). That tracking code file you put on your site? It’s not a static Javascript file. It’s a PHP script in disguise. As are countless Javascript files on the internet. There are a lot of advantages to doing it that way, but unfortunately if you want to move that to a CDN, it has to change to a fully static file.

The reason we must maintain this file as a dynamic script is for backwards compatibility purposes. With our old school tracking code, every site had their own unique URL, e.g. static.getclicky.com/123.js. 123 in that case is the site ID, and with the magic of Apache’s mod_rewrite, it secretly points instead to /js.php?site_id=123. When the site ID is attached to the URL like this, we’re only adding one line of code (to automatically call the clicky.init( SITE_ID ) function), so that’s nothing fancy – but no CDN will support something like that.

Also, we have our self-hosted tracking code folks to think about. To minimize DNS lookups, static.getclicky.com is the domain used both for hosting the tracking code, as well as logging the incoming traffic data for your web site. If we just moved static.getclicky.com to a normal CDN, everyone who is self-hosting the code would no longer have their traffic logged unless they updated their tracking code (which will be pointing incoming traffic to a new sub-domain).

It would be literally impossible to get everyone who uses our service to update to the new tracking code (both for self-hosted folks, as well as old school folks). To maintain backwards compatibility for every single one of our users, we must have more control.

If we had it to do all over again, we would have used a fully static file from the beginning and all would be well. Unfortunately, we not do have access to a time machine.

And your point is…?

We’re building our own stinking CDN. We’re going to lease a few servers in key parts of the world – One in central or east US, one in Europe, one in Hong Kong, and one in Australia. And we’re going to use Geo-Directional DNS so that a request for static.getclicky.com will resolve to the server that is closest to you geographically.

By building our own CDN, we can use SSL with our own domain, we can use dynamic scripts, and we can redirect incoming traffic log requests to the correct servers. All of these are critical components of a CDN for us, and none of them are met with any of the existing CDNs.

We need your help

We’re very excited about this, but we need your help. We build our own servers and host them locally, but for this project, that’s out of the question.

We need recommendations for companies in Central/East US, Europe, Hong Kong, and Australia, that lease servers with the following requirements:

• Debian Linux 4 or 5 with full root access
• Dedicated IP address
• 4+ GB RAM
• 2+ TB monthly bandwidth (with reasonable fee to upgrade to a higher level)
• $250 USD or less per month
• Decent tech support
• Web site in English

If you have experience with any such companies in the markets listed above, please leave a comment here. We want to have this fully deployed by the end of the month, so your help is very much appreciated.

Note – for Europe, we’re already leaning towards Hetzner, which is located in Germany. We’ve read quite a few good things about them, but that doesn’t mean we won’t go with someone else if you think someone else is better.

We’ve been tracking visitor organization data for a long time, but if you wanted to learn more about any of them, you had to research that on your own. No more!

We have integrated data from jigsaw.com that will display more information about most organizations in the US, Canada, and the UK. It doesn’t look like they have any data about companies outside of these countries, but I’m not entirely sure.

Anyways, you can access this new data from two different areas. The first one is the global organizations report. You will see a new magnifying glass (search) icon next to each organization now:

When you click the icon you will see a list of potential matches:

Clicking on one will display the following data points about the company:

When viewing details on an individual visitor, next to where their organization is listed is a new link, Lookup:

It works the exact same way, click to find potential matches and then click a match to view more details:

We hope you like it. We wish this data was available on a more global basis but this is the best we can do for now. Half of our customers are in the supported countries so we know at least half of you will like it

Clicky launched on November 28, 2006. I can’t believe it’s been three years, but the calendars don’t lie. Clicky is now tracking over 155,000 web sites, powered by over 30 servers chugging along in the background. We never expected it to get this big. We greatly appreciate everyone who has supported us throughout the years and told your friends and colleagues about our service – Clicky has spread purely by word of mouth and that’s the best kind of marketing anyone can ask for. Thank you for everything.