librelist archives

Re: [cool.io] Trying to hack together a Rainbows/Coolio project

From:

Tim Inman

Date:

2010-12-31 @ 10:02

OK, Thanks, Eric.

I have the part about the config file sorted out now, but I'm still a
little unclear.  Here's my config.ru now:
<pre>
require 'cool.io'
HOST = 'localhost'
PORT = 4321

class EchoServerConnection < Cool.io::TCPSocket
  def on_connect
    puts "#{remote_addr}:#{remote_port} connected"
  end

  def on_close
    puts "#{remote_addr}:#{remote_port} disconnected"
  end

  def on_read(data)
    write data
  end
end

server = Cool.io::TCPServer.new(HOST, PORT, EchoServerConnection)
server.attach(Cool.io::Loop.default)

Cool.io::Loop.default.run

app = proc do |env|
	[ 200, {'Content-Type' => 'text/html'}, [ "hello world"] ]
end

run app
</pre>

I think what I am unclear on is:
> Rainbows! already starts a Cool.io loop for you (the default), so
> your app can just attach to that and forget about running it.

At first, I thougt you meant to omit "Cool.io::Loop.default.run"; If I
do that I get "hello world" on port 8080, but no Echo server
connecting or disconnecting on either port.

Then I thought you may have meant to omit "run app", but then I don't
get 'hello world'
Can you spell out 'your app can just attach to that and forget about
running it.'?
Also, I know rack by default serves on 0.0.0.0:8080, do I keep coolio
on localhost:4321?

Thanks!

Tim



On Fri, Dec 31, 2010 at 4:23 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Tim Inman <tim.inman@gmail.com> wrote:
>> I have been able to come up with an html response in Rainbows with
>> rack, or, alternatively, a Coolio server that outputs to the terminal
>> when I visit localhost:4321, but so far I haven't been able to get it
>> all working together. Here's one of the ways I was trying earlier:
>> https://gist.github.com/759864 .
>>
>> If I run it with the vanilla "rainbows" command, I get "hello world"
>> at localhost:8080. If I use the "-c" option: "rainbows -c
>> coolio_config.rb", I get the coolio server which will connect and
>> disconnect and output to the terminal.
>
> Rainbows! already starts a Cool.io loop for you (the default), so
> your app can just attach to that and forget about running it.
>
>> I'm assuming I can use Cool.io to get Ranbows to output something to
>> html.  Is that really possible? What should I be doing differently?
>
> Your coolio_config.rb (which you pass as "-c") can just be:
>
> --------------- 8< -----------------
> Rainbows! do
>  use :Coolio
> end
> --------------- 8< -----------------
>
> You can stick the EchoServerConnection class in $APP_ROOT/lib
> or directly in config.ru, and just attach it (no need to run
> the default loop, Rainbows! is already running it).
>
> Also, the response body (last element of the array) should
> be an Array itself:
>
> --------------- 8< -----------------
>  app = proc do |env|
>    [ 200, {'Content-Type' => 'text/html'}, [ "hello world" ] ]
>  end
> --------------- 8< -----------------
>
> --
> Eric Wong
>

Tim Inman <tim.inman@gmail.com> wrote:
> OK, Thanks, Eric.
> 
> I have the part about the config file sorted out now, but I'm still a
> little unclear.  Here's my config.ru now:
> <pre>
> require 'cool.io'
> HOST = 'localhost'
> PORT = 4321
> 
> class EchoServerConnection < Cool.io::TCPSocket
>   def on_connect
>     puts "#{remote_addr}:#{remote_port} connected"
>   end
> 
>   def on_close
>     puts "#{remote_addr}:#{remote_port} disconnected"
>   end
> 
>   def on_read(data)
>     write data
>   end
> end
> 
> server = Cool.io::TCPServer.new(HOST, PORT, EchoServerConnection)
> server.attach(Cool.io::Loop.default)
> 
> Cool.io::Loop.default.run
> 
> app = proc do |env|
> 	[ 200, {'Content-Type' => 'text/html'}, [ "hello world"] ]
> end
> 
> run app
> </pre>
> 
> I think what I am unclear on is:
> > Rainbows! already starts a Cool.io loop for you (the default), so
> > your app can just attach to that and forget about running it.
> 
> At first, I thougt you meant to omit "Cool.io::Loop.default.run"; If I
> do that I get "hello world" on port 8080, but no Echo server
> connecting or disconnecting on either port.

omitting "Cool.io::Loop.default.run" is exactly what I meant.

> Then I thought you may have meant to omit "run app", but then I don't
> get 'hello world'
> Can you spell out 'your app can just attach to that and forget about
> running it.'?
> Also, I know rack by default serves on 0.0.0.0:8080, do I keep coolio
> on localhost:4321?

Rainbows! will bind to 0.0.0.0:8080 by default and its internal Cool.io
listener will be attached to that address, but Cool.io can have multiple
listeners with your echo server being another.

I've verified the following two config files work with Ruby 1.9.2
(I added Rack::ContentLength to pass Rack::Lint and to make life easier
for keepalive clients).

# -----------------------  config.ru ------------------------
HOST = 'localhost'
PORT = 4321

class EchoServerConnection < Cool.io::TCPSocket
  def on_connect
    puts "#{remote_addr}:#{remote_port} connected"
  end

  def on_close
    puts "#{remote_addr}:#{remote_port} disconnected"
  end

  def on_read(data)
    write data
  end
end
server = Cool.io::TCPServer.new(HOST, PORT, EchoServerConnection)
server.attach(Cool.io::Loop.default)

puts "Echo server listening on #{HOST}:#{PORT}"
use Rack::ContentLength
app = proc do |env|
  [ 200, {'Content-Type' => 'text/html'}, [ "hello world" ] ]
end

run app

-------------------- rainbows.conf.rb --------------------
Rainbows! do
  use :Coolio
end

-- 
Eric Wong

Awesome. I've been looking for a comprehensive example like this for awhile.
I might try to slap it up on http://coolio.github.com

On Fri, Dec 31, 2010 at 9:57 AM, Eric Wong <normalperson@yhbt.net> wrote:

> Tim Inman <tim.inman@gmail.com> wrote:
> > OK, Thanks, Eric.
> >
> > I have the part about the config file sorted out now, but I'm still a
> > little unclear.  Here's my config.ru now:
> > <pre>
> > require 'cool.io'
> > HOST = 'localhost'
> > PORT = 4321
> >
> > class EchoServerConnection < Cool.io::TCPSocket
> >   def on_connect
> >     puts "#{remote_addr}:#{remote_port} connected"
> >   end
> >
> >   def on_close
> >     puts "#{remote_addr}:#{remote_port} disconnected"
> >   end
> >
> >   def on_read(data)
> >     write data
> >   end
> > end
> >
> > server = Cool.io::TCPServer.new(HOST, PORT, EchoServerConnection)
> > server.attach(Cool.io::Loop.default)
> >
> > Cool.io::Loop.default.run
> >
> > app = proc do |env|
> >       [ 200, {'Content-Type' => 'text/html'}, [ "hello world"] ]
> > end
> >
> > run app
> > </pre>
> >
> > I think what I am unclear on is:
> > > Rainbows! already starts a Cool.io loop for you (the default), so
> > > your app can just attach to that and forget about running it.
> >
> > At first, I thougt you meant to omit "Cool.io::Loop.default.run"; If I
> > do that I get "hello world" on port 8080, but no Echo server
> > connecting or disconnecting on either port.
>
> omitting "Cool.io::Loop.default.run" is exactly what I meant.
>
> > Then I thought you may have meant to omit "run app", but then I don't
> > get 'hello world'
> > Can you spell out 'your app can just attach to that and forget about
> > running it.'?
> > Also, I know rack by default serves on 0.0.0.0:8080, do I keep coolio
> > on localhost:4321?
>
> Rainbows! will bind to 0.0.0.0:8080 by default and its internal Cool.io
> listener will be attached to that address, but Cool.io can have multiple
> listeners with your echo server being another.
>
> I've verified the following two config files work with Ruby 1.9.2
> (I added Rack::ContentLength to pass Rack::Lint and to make life easier
> for keepalive clients).
>
> # -----------------------  config.ru ------------------------
> HOST = 'localhost'
> PORT = 4321
>
> class EchoServerConnection < Cool.io::TCPSocket
>  def on_connect
>    puts "#{remote_addr}:#{remote_port} connected"
>  end
>
>  def on_close
>    puts "#{remote_addr}:#{remote_port} disconnected"
>  end
>
>  def on_read(data)
>    write data
>  end
> end
> server = Cool.io::TCPServer.new(HOST, PORT, EchoServerConnection)
> server.attach(Cool.io::Loop.default)
>
> puts "Echo server listening on #{HOST}:#{PORT}"
> use Rack::ContentLength
> app = proc do |env|
>  [ 200, {'Content-Type' => 'text/html'}, [ "hello world" ] ]
> end
>
> run app
>
> -------------------- rainbows.conf.rb --------------------
> Rainbows! do
>  use :Coolio
> end
>
> --
> Eric Wong
>



-- 
Tony Arcieri
Medioh! Kudelski

So I probably need a bit of help from Eric on this. I'd like to create an
example which shows communication between a Cool.io TCPServer and a client
making an HTTP request through Rainbows!

I'd like to keep things really simple, perhaps an example that shows how to
post a message to all clients that are actively connected through a standard
TCP server rather than an HTTP server. Sort of like irccat for HTTP I guess?
Or a very simple and possibly lossy pubsub system.

How should the TCPServer communicate to the Rainbows! clients? Through an
AsyncWatcher? Sorry I can't be more helpful here but I spent my time working
on this Coolio In Rainbows image instead:

http://coolio.github.com/images/coolio-in-rainbows.jpg

On Fri, Dec 31, 2010 at 3:17 PM, Tony Arcieri <tony.arcieri@medioh.com>wrote:

> Awesome. I've been looking for a comprehensive example like this for
> awhile. I might try to slap it up on http://coolio.github.com
>
>
> On Fri, Dec 31, 2010 at 9:57 AM, Eric Wong <normalperson@yhbt.net> wrote:
>
>> Tim Inman <tim.inman@gmail.com> wrote:
>> > OK, Thanks, Eric.
>> >
>> > I have the part about the config file sorted out now, but I'm still a
>> > little unclear.  Here's my config.ru now:
>> > <pre>
>> > require 'cool.io'
>> > HOST = 'localhost'
>> > PORT = 4321
>> >
>> > class EchoServerConnection < Cool.io::TCPSocket
>> >   def on_connect
>> >     puts "#{remote_addr}:#{remote_port} connected"
>> >   end
>> >
>> >   def on_close
>> >     puts "#{remote_addr}:#{remote_port} disconnected"
>> >   end
>> >
>> >   def on_read(data)
>> >     write data
>> >   end
>> > end
>> >
>> > server = Cool.io::TCPServer.new(HOST, PORT, EchoServerConnection)
>> > server.attach(Cool.io::Loop.default)
>> >
>> > Cool.io::Loop.default.run
>> >
>> > app = proc do |env|
>> >       [ 200, {'Content-Type' => 'text/html'}, [ "hello world"] ]
>> > end
>> >
>> > run app
>> > </pre>
>> >
>> > I think what I am unclear on is:
>> > > Rainbows! already starts a Cool.io loop for you (the default), so
>> > > your app can just attach to that and forget about running it.
>> >
>> > At first, I thougt you meant to omit "Cool.io::Loop.default.run"; If I
>> > do that I get "hello world" on port 8080, but no Echo server
>> > connecting or disconnecting on either port.
>>
>> omitting "Cool.io::Loop.default.run" is exactly what I meant.
>>
>> > Then I thought you may have meant to omit "run app", but then I don't
>> > get 'hello world'
>> > Can you spell out 'your app can just attach to that and forget about
>> > running it.'?
>> > Also, I know rack by default serves on 0.0.0.0:8080, do I keep coolio
>> > on localhost:4321?
>>
>> Rainbows! will bind to 0.0.0.0:8080 by default and its internal Cool.io
>> listener will be attached to that address, but Cool.io can have multiple
>> listeners with your echo server being another.
>>
>> I've verified the following two config files work with Ruby 1.9.2
>> (I added Rack::ContentLength to pass Rack::Lint and to make life easier
>> for keepalive clients).
>>
>> # -----------------------  config.ru ------------------------
>> HOST = 'localhost'
>> PORT = 4321
>>
>> class EchoServerConnection < Cool.io::TCPSocket
>>  def on_connect
>>    puts "#{remote_addr}:#{remote_port} connected"
>>  end
>>
>>  def on_close
>>    puts "#{remote_addr}:#{remote_port} disconnected"
>>  end
>>
>>  def on_read(data)
>>    write data
>>  end
>> end
>> server = Cool.io::TCPServer.new(HOST, PORT, EchoServerConnection)
>> server.attach(Cool.io::Loop.default)
>>
>> puts "Echo server listening on #{HOST}:#{PORT}"
>> use Rack::ContentLength
>> app = proc do |env|
>>  [ 200, {'Content-Type' => 'text/html'}, [ "hello world" ] ]
>> end
>>
>> run app
>>
>> -------------------- rainbows.conf.rb --------------------
>> Rainbows! do
>>  use :Coolio
>> end
>>
>> --
>> Eric Wong
>>
>
>
>
> --
> Tony Arcieri
> Medioh! Kudelski
>



-- 
Tony Arcieri
Medioh! Kudelski

Tony Arcieri <tony.arcieri@medioh.com> wrote:
> So I probably need a bit of help from Eric on this. I'd like to create an
> example which shows communication between a Cool.io TCPServer and a client
> making an HTTP request through Rainbows!
> 
> I'd like to keep things really simple, perhaps an example that shows how to
> post a message to all clients that are actively connected through a standard
> TCP server rather than an HTTP server. Sort of like irccat for HTTP I guess?
> Or a very simple and possibly lossy pubsub system.

I'll probably need to add the "async.callback" support that James Tucker
did for EventMachine + async_sinatra to the Coolio support modules in
Rainbows! to get more flexibility for this.

Currently, Coolio (and before, Rev) is great with Rainbows! for hosting
web services that can handle many simultaneous clients but only have
local, non-network I/O dependencies (e.g. SQLite, TokyoCabinet, DBM,
GDBM, TDB[1], or static files).

Most one-off applications I create for myself tend to be in this
category of having only non-network dependencies and only have 1-4
simultaneously connections (but I've simulated *thousands* :).

For a pubsub system, the slightly trickier part is also getting all the
worker processes coordinated for a publish to all subscribers because
(ideally) the subcribers are spread across different workers on a
multi-core system given the GVL in Ruby 1.9.

Coolio's inotify support can be used for this, I imagine.

This would be trivial with Zbatery[2] where worker_processes==1, always.

Coolio can (and always has) been able to proxy response bodies (using
Rainbows::DevFdResponse on Linux) if they're pipe/socket objects
asynchronously, but it's somewhat awkward and difficult to use (the
integration test suite has some examples, but they're more contrived
than anything).

> How should the TCPServer communicate to the Rainbows! clients? Through an
> AsyncWatcher? Sorry I can't be more helpful here but I spent my time working
> on this Coolio In Rainbows image instead:
> 
> http://coolio.github.com/images/coolio-in-rainbows.jpg

Heh.

[1] - http://bogomips.org/ruby-tdb/ shameless plug
[2] - http://zbatery.bogomip.org/ (Rainbows! without fork())

-- 
Eric Wong

Eric Wong <normalperson@yhbt.net> wrote:
> Tony Arcieri <tony.arcieri@medioh.com> wrote:
> > So I probably need a bit of help from Eric on this. I'd like to create an
> > example which shows communication between a Cool.io TCPServer and a client
> > making an HTTP request through Rainbows!
> > 
> > I'd like to keep things really simple, perhaps an example that shows how to
> > post a message to all clients that are actively connected through a standard
> > TCP server rather than an HTTP server. Sort of like irccat for HTTP I guess?
> > Or a very simple and possibly lossy pubsub system.
> 
> I'll probably need to add the "async.callback" support that James Tucker
> did for EventMachine + async_sinatra to the Coolio support modules in
> Rainbows! to get more flexibility for this.

I've been working on this, but unfortunately there's nothing like
EM::Deferrable with callback/errback functionality.  I guess the
on_close callback comes close for IO, but nothing comparable for
timers.

This basically means Rainbows! can only reliably do a one-shot write of
an HTTP response body through this interface and not trickle updates to
a client (at least not through this interface, with Fibers it's easy)
(at least not with keepalive support).

On the other hand, being able to do a one-shot write of an HTTP response
body is already quite powerful and probably cheaper than Fibers.

-- 
Eric Wong

On Tue, Jan 4, 2011 at 3:25 PM, Eric Wong <normalperson@yhbt.net> wrote:

> For a pubsub system, the slightly trickier part is also getting all the
> worker processes coordinated for a publish to all subscribers because
> (ideally) the subcribers are spread across different workers on a
> multi-core system given the GVL in Ruby 1.9.


Then there's the larger matter of a distributed system where all of your
workers aren't on the same host. I was working on a Rack middleware to do
this called Pistachio, which provides a simple HTTP long-polling API which
is backed by Redis:

https://github.com/tarcieri/pistachio

It'd be neat if I could get this going asynchronously with Cool.io, although
clearly the POLS here would just be to use EventMachine since there's
already em-redis and there's presently no asynchronous Redis adapter for
Cool.io.

I was thinking of the simple pubsub system I described more as a sort of
getting started example of how people could use Cool.io in Rainbows to
implement async handlers that are waiting indefinitely on an external event
to happen before returning an HTTP response rather than a fully fledged
production-ready pubsub system, and really just something simple but
slightly less trivial than the example you gave above that I could slap up
on http://coolio.github.com

If you have any ideas for such examples that could be done with Cool.io and
Rainbows today, I'd love to hear them.

-- 
Tony Arcieri
Medioh! Kudelski

Nope. It still didn't work for me. I checked it and double-checked it.  It
serves the 'hello world', and even puts "Echo server listening on
#{HOST}:#{PORT}", but it's not really working because it's not outputting
all of the connections and disconnections to the terminal.

On a whim *I swapped every 'Coolio' or 'Cool.io' in the program to Rev*, and
that made it work!  There must be some incompatibility yet in there.

Thanks for your library and all your help.

Tim




On Fri, Dec 31, 2010 at 4:57 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Tim Inman <tim.inman@gmail.com> wrote:
>> OK, Thanks, Eric.
>>
>> I have the part about the config file sorted out now, but I'm still a
>> little unclear.  Here's my config.ru now:
>> <pre>
>> require 'cool.io'
>> HOST = 'localhost'
>> PORT = 4321
>>
>> class EchoServerConnection < Cool.io::TCPSocket
>>   def on_connect
>>     puts "#{remote_addr}:#{remote_port} connected"
>>   end
>>
>>   def on_close
>>     puts "#{remote_addr}:#{remote_port} disconnected"
>>   end
>>
>>   def on_read(data)
>>     write data
>>   end
>> end
>>
>> server = Cool.io::TCPServer.new(HOST, PORT, EchoServerConnection)
>> server.attach(Cool.io::Loop.default)
>>
>> Cool.io::Loop.default.run
>>
>> app = proc do |env|
>>       [ 200, {'Content-Type' => 'text/html'}, [ "hello world"] ]
>> end
>>
>> run app
>> </pre>
>>
>> I think what I am unclear on is:
>> > Rainbows! already starts a Cool.io loop for you (the default), so
>> > your app can just attach to that and forget about running it.
>>
>> At first, I thougt you meant to omit "Cool.io::Loop.default.run"; If I
>> do that I get "hello world" on port 8080, but no Echo server
>> connecting or disconnecting on either port.
>
> omitting "Cool.io::Loop.default.run" is exactly what I meant.
>
>> Then I thought you may have meant to omit "run app", but then I don't
>> get 'hello world'
>> Can you spell out 'your app can just attach to that and forget about
>> running it.'?
>> Also, I know rack by default serves on 0.0.0.0:8080, do I keep coolio
>> on localhost:4321?
>
> Rainbows! will bind to 0.0.0.0:8080 by default and its internal Cool.io
> listener will be attached to that address, but Cool.io can have multiple
> listeners with your echo server being another.
>
> I've verified the following two config files work with Ruby 1.9.2
> (I added Rack::ContentLength to pass Rack::Lint and to make life easier
> for keepalive clients).
>
> # -----------------------  config.ru ------------------------
> HOST = 'localhost'
> PORT = 4321
>
> class EchoServerConnection < Cool.io::TCPSocket
>  def on_connect
>    puts "#{remote_addr}:#{remote_port} connected"
>  end
>
>  def on_close
>    puts "#{remote_addr}:#{remote_port} disconnected"
>  end
>
>  def on_read(data)
>    write data
>  end
> end
> server = Cool.io::TCPServer.new(HOST, PORT, EchoServerConnection)
> server.attach(Cool.io::Loop.default)
>
> puts "Echo server listening on #{HOST}:#{PORT}"
> use Rack::ContentLength
> app = proc do |env|
>  [ 200, {'Content-Type' => 'text/html'}, [ "hello world" ] ]
> end
>
> run app
>
> -------------------- rainbows.conf.rb --------------------
> Rainbows! do
>  use :Coolio
> end
>
> --
> Eric Wong
>

Tim Inman <tim.inman@gmail.com> wrote:
> On a whim *I swapped every 'Coolio' or 'Cool.io' in the program to Rev*, and
> that made it work!  There must be some incompatibility yet in there.

Which version of Rainbows! did you try?  2.1.0 should use Coolio
everywhere and only keep Rev as a backwards compatibility option.

-- 
Eric Wong

Yeah, that must be it.  I'd just done a vanilla 'sudo gem install' and ended
up with 1.3.7.

Tim


On Fri, Dec 31, 2010 at 7:56 PM, Eric Wong <normalperson@yhbt.net> wrote:

> Tim Inman <tim.inman@gmail.com> wrote:
> > On a whim *I swapped every 'Coolio' or 'Cool.io' in the program to Rev*,
> and
> > that made it work!  There must be some incompatibility yet in there.
>
> Which version of Rainbows! did you try?  2.1.0 should use Coolio
> everywhere and only keep Rev as a backwards compatibility option.
>
> --
> Eric Wong
>