librelist archives

« back to archive

KadOH DHT experimental twitter-like application

KadOH DHT experimental twitter-like application

From:
Alexandre Lachèze
Date:
2012-06-25 @ 18:54
Hi everyone,

As we announced earlier (“Introducing KadOH, a Javascript framework
for P2P based distributed applications”), we were busy building a
prototype application on top of KadOH. Now, it’s time to present it !

So, KadOH-twitter is an application that imitates basic twitter
functionalities, but in which all the data are stored in a web DHT
(aka in users’ browser) and communication between nodes uses XMPP!

Okay, that sounds great, but we launch this just as an experiment
during a limited period of time (say 1 week if all runs well). We want
indeed to measure how our DHT behaves on top of XMPP when used by real
users in a real application. So don’t expect to have a completed,
secured and optimized twitter-like service, neither totally reliable
but, please, make some noise on it : let’s tweet !

http://twitter.kadoh.fr.nf/

Don’t hesitate to use your smartphone since we are interested of
seeing performance on this kind of device. All your actions will be
anonymously logged in a database and we’ll later compute statistics
based on them.

You can “follow” us :
http://twitter.kadoh.fr.nf/#author/alex
http://twitter.kadoh.fr.nf/#author/pierre
or check what’s going on the dedicated #webp2p hashtag :
http://twitter.kadoh.fr.nf/#hashtag/webp2p

Hackers warning :
The application we built is not designed to be robust to attacks, and
since all is open, there is no merit to hack on it. So please don’t
feel like breaking our experiment, this will just be a waste of time
for us..

Technical details:
  - we did this application in a hurry, because we need some results
rapidly. So you can check the code here :
https://github.com/alexstrat/kadoh-twitter, but it’s done quick and
dirty.
  - we tried in this project to bring a REST-like interface to our DHT
and, as proof of that, we used BackboneJS directly to deal with the
models and collections stored on the DHT. By the way, we found this
REST/DHT interplay interesting and we think there is something
theoretically we’d like to explore later regarding data-structures on
DHTs..
  - our logging service uses the great project cube. Every clients
runs a reporter service sending informations to a cube collector
running on an EC2 instance that stores all the informations on a Mogo
Database (see https://github.com/square/cube)
  - to provide a direct access to the application, we used XMPP
anonymous login : don’t need to enter a Jabber or GMail account
credentials.
  - to give a bit of reliability to our DHT, we run continuously 30
bots. Maybe we don’t need so much : we hope our result will show us
how many are needed to ensure a certain degree of reliability.

Cheers,

Alexandre L. and Pierre G.

Re: KadOH DHT experimental twitter-like application

From:
Pierre Guilleminot
Date:
2012-06-25 @ 19:48
Hey !

Just to clarify from the first feedbacks we got : we do not support
yet following, rt and mentions.

And keep in mind that the idea for us is not to have a real "twitter"
application for now, but to log many routing informations thanks to
all of you that are tweeting/chatting.

Cheers :)

On Mon, Jun 25, 2012 at 8:54 PM, Alexandre Lachèze
<alexandre.lacheze@gmail.com> wrote:
> Hi everyone,
>
> As we announced earlier (“Introducing KadOH, a Javascript framework
> for P2P based distributed applications”), we were busy building a
> prototype application on top of KadOH. Now, it’s time to present it !
>
> So, KadOH-twitter is an application that imitates basic twitter
> functionalities, but in which all the data are stored in a web DHT
> (aka in users’ browser) and communication between nodes uses XMPP!
>
> Okay, that sounds great, but we launch this just as an experiment
> during a limited period of time (say 1 week if all runs well). We want
> indeed to measure how our DHT behaves on top of XMPP when used by real
> users in a real application. So don’t expect to have a completed,
> secured and optimized twitter-like service, neither totally reliable
> but, please, make some noise on it : let’s tweet !
>
> http://twitter.kadoh.fr.nf/
>
> Don’t hesitate to use your smartphone since we are interested of
> seeing performance on this kind of device. All your actions will be
> anonymously logged in a database and we’ll later compute statistics
> based on them.
>
> You can “follow” us :
> http://twitter.kadoh.fr.nf/#author/alex
> http://twitter.kadoh.fr.nf/#author/pierre
> or check what’s going on the dedicated #webp2p hashtag :
> http://twitter.kadoh.fr.nf/#hashtag/webp2p
>
> Hackers warning :
> The application we built is not designed to be robust to attacks, and
> since all is open, there is no merit to hack on it. So please don’t
> feel like breaking our experiment, this will just be a waste of time
> for us..
>
> Technical details:
>  - we did this application in a hurry, because we need some results
> rapidly. So you can check the code here :
> https://github.com/alexstrat/kadoh-twitter, but it’s done quick and
> dirty.
>  - we tried in this project to bring a REST-like interface to our DHT
> and, as proof of that, we used BackboneJS directly to deal with the
> models and collections stored on the DHT. By the way, we found this
> REST/DHT interplay interesting and we think there is something
> theoretically we’d like to explore later regarding data-structures on
> DHTs..
>  - our logging service uses the great project cube. Every clients
> runs a reporter service sending informations to a cube collector
> running on an EC2 instance that stores all the informations on a Mogo
> Database (see https://github.com/square/cube)
>  - to provide a direct access to the application, we used XMPP
> anonymous login : don’t need to enter a Jabber or GMail account
> credentials.
>  - to give a bit of reliability to our DHT, we run continuously 30
> bots. Maybe we don’t need so much : we hope our result will show us
> how many are needed to ensure a certain degree of reliability.
>
> Cheers,
>
> Alexandre L. and Pierre G.

Re: [webp2p] Re: KadOH DHT experimental twitter-like application

From:
Sean Lynch
Date:
2012-06-25 @ 19:59
On 6/25/12 12:48 PM, Pierre Guilleminot wrote:
> Okay, that sounds great, but we launch this just as an experiment
> >during a limited period of time (say 1 week if all runs well). We want
> >indeed to measure how our DHT behaves on top of XMPP when used by real
> >users in a real application. So don’t expect to have a completed,
> >secured and optimized twitter-like service, neither totally reliable
> >but, please, make some noise on it : let’s tweet !
> >
> >http://twitter.kadoh.fr.nf/
> >
I'd recommend against using "twitter" or "tweet" in any terminology 
related to this service. You'll just end up having to change it later, 
and the longer you use it the more work that will be.

Re: [webp2p] Re: KadOH DHT experimental twitter-like application

From:
Rich Jones
Date:
2012-06-26 @ 02:26
Absolutely fantastic!

Been playing with this all day, nice tweeting at y'all!

I encountered a few bugs but couldn't record them properly - it'd be cool
if you could put some logging functionality in this so we could report bugs
in a useful fashion.

Interesting use of Backbone - did you have to modify the library at all, or
did it work out of the box? I have played with Backbone before but found it
to be a bit cumbersome and perhaps unnecessary, though I'd like to give it
another go.

I love that you guys are doing this in such a data-driven fashion, too, I
really can't wait to see your results.

Anyway, very cool stuff! It'd be sweet if you stripped out the
transport/data from the application as much as possible, so we could reuse
this, or something like it, as a template for other distributed
applications and contribute back bug fixes and new features (accounts,
session management, File API access, etc.)

-- 
Rich Jones
Director, Gun.io

Re: [webp2p] Re: KadOH DHT experimental twitter-like application

From:
Alexandre Lachèze
Date:
2012-06-27 @ 16:09
2012/6/26 Rich Jones <rich@gun.io>:
> Absolutely fantastic!
>
> Been playing with this all day, nice tweeting at y'all!
>
Thank you for your contribution !! :)

> I encountered a few bugs but couldn't record them properly - it'd be cool if
> you could put some logging functionality in this so we could report bugs in
> a useful fashion.

Yes, you're right we could have done that, but debugging is really
hard on kadoh since it's distributed. The existing debug-ui
(https://github.com/jinroh/kadoh/wiki/Debug-UI) would have been a
solution, but we didn't take time to adapt it for twitter..
Also, keep in mind that the goal for us was to do an attractive
application to get some activity and then extract performance data
from it. So, if there are little bugs, it is not so bad.. apart from
the user's point of view.

> Interesting use of Backbone - did you have to modify the library at all, or
> did it work out of the box? I have played with Backbone before but found it
> to be a bit cumbersome and perhaps unnecessary, though I'd like to give it
> another go.

Actually the only tweak we did to backbone is overriding the sync
(http://backbonejs.org/#Sync) method, that determines the way to
communicate with the server in a traditional application. But backbone
is designed to let devellopers override this, so it's not a real
tweak.
Personnaly I got a different point of view on backbone. This is the
second time I build an application with backbone and I found it leight
and non-obstrusive comparing to other similare frameworks (say
SproutCore, ..). Concerning the necessity of it, it's true that you
can do without but you need to think the architecture hard before
coding to let it be extensible and reusable. Backbone gives you this
architecture and encourages you to do it the right way without
thinking too much. That fits pretty well for prototype application
done in a hurry like ours !

> I love that you guys are doing this in such a data-driven fashion, too, I
> really can't wait to see your results.
>
> Anyway, very cool stuff! It'd be sweet if you stripped out the
> transport/data from the application as much as possible, so we could reuse
> this, or something like it, as a template for other distributed applications
> and contribute back bug fixes and new features (accounts, session
> management, File API access, etc.)

That's in our plan. We want to extract what it reusable from our app
and concepts to let it be a more advanced framework than kadoh to
build applications on top of kadoh. We'll let you know..

Alexandre

Re: KadOH DHT experimental twitter-like application

From:
Alexandre Lachèze
Date:
2012-06-27 @ 15:45
Hi all, again,

Like some of you probably noticed "something went (globally) wrong" in
our experiment on Tuesday early in the morning (CET). This was due to
the lost of our 30 bots that ran out of memory on our overloaded
Amazon EC2 instance.. Despite the fact that our system is distributed,
the lost of these 30 bots and the 2 bootstraps was hardly recoverable.
So we decided to relaunch our experiment from scratch...

But, because we want make you come back and tweet, we put some pieces
of improvement :
  - you can now put mentions in your tweet and going to the user's
page lets you see the tweets from the user and the ones that mention
the user. Nice for interaction!
  - the latest tweets are now really the latest tweets ! No empty page
anymore when loading latest tweets..
  - security improved and limitation to 140 characters..
  - we tune differently the DHT parameters to improve reliability when
bots fails
  - little bug fixes..

So, please again, make some noise ! Let's tweet ! Re-ask your friend
and colleague to tell their minds on twitter.kadoh.fr.nf !!

We are so sorry to have lost all of your tweets, but we thank you very
much for your past and future participation ! We are sure we will get
some interesting results !

Alexandre and Pierre


2012/6/25 Pierre Guilleminot <pierre.guilleminot@gmail.com>:
> Hey !
>
> Just to clarify from the first feedbacks we got : we do not support
> yet following, rt and mentions.
>
> And keep in mind that the idea for us is not to have a real "twitter"
> application for now, but to log many routing informations thanks to
> all of you that are tweeting/chatting.
>
> Cheers :)
>
> On Mon, Jun 25, 2012 at 8:54 PM, Alexandre Lachèze
> <alexandre.lacheze@gmail.com> wrote:
>> Hi everyone,
>>
>> As we announced earlier (“Introducing KadOH, a Javascript framework
>> for P2P based distributed applications”), we were busy building a
>> prototype application on top of KadOH. Now, it’s time to present it !
>>
>> So, KadOH-twitter is an application that imitates basic twitter
>> functionalities, but in which all the data are stored in a web DHT
>> (aka in users’ browser) and communication between nodes uses XMPP!
>>
>> Okay, that sounds great, but we launch this just as an experiment
>> during a limited period of time (say 1 week if all runs well). We want
>> indeed to measure how our DHT behaves on top of XMPP when used by real
>> users in a real application. So don’t expect to have a completed,
>> secured and optimized twitter-like service, neither totally reliable
>> but, please, make some noise on it : let’s tweet !
>>
>> http://twitter.kadoh.fr.nf/
>>
>> Don’t hesitate to use your smartphone since we are interested of
>> seeing performance on this kind of device. All your actions will be
>> anonymously logged in a database and we’ll later compute statistics
>> based on them.
>>
>> You can “follow” us :
>> http://twitter.kadoh.fr.nf/#author/alex
>> http://twitter.kadoh.fr.nf/#author/pierre
>> or check what’s going on the dedicated #webp2p hashtag :
>> http://twitter.kadoh.fr.nf/#hashtag/webp2p
>>
>> Hackers warning :
>> The application we built is not designed to be robust to attacks, and
>> since all is open, there is no merit to hack on it. So please don’t
>> feel like breaking our experiment, this will just be a waste of time
>> for us..
>>
>> Technical details:
>>  - we did this application in a hurry, because we need some results
>> rapidly. So you can check the code here :
>> https://github.com/alexstrat/kadoh-twitter, but it’s done quick and
>> dirty.
>>  - we tried in this project to bring a REST-like interface to our DHT
>> and, as proof of that, we used BackboneJS directly to deal with the
>> models and collections stored on the DHT. By the way, we found this
>> REST/DHT interplay interesting and we think there is something
>> theoretically we’d like to explore later regarding data-structures on
>> DHTs..
>>  - our logging service uses the great project cube. Every clients
>> runs a reporter service sending informations to a cube collector
>> running on an EC2 instance that stores all the informations on a Mogo
>> Database (see https://github.com/square/cube)
>>  - to provide a direct access to the application, we used XMPP
>> anonymous login : don’t need to enter a Jabber or GMail account
>> credentials.
>>  - to give a bit of reliability to our DHT, we run continuously 30
>> bots. Maybe we don’t need so much : we hope our result will show us
>> how many are needed to ensure a certain degree of reliability.
>>
>> Cheers,
>>
>> Alexandre L. and Pierre G.