librelist archives

« back to archive

RForum -> LibreForum

RForum -> LibreForum

From:
Trans
Date:
2010-01-20 @ 18:05
So I like Librelist, but I also like using a web-interface instead of
my inbox for interacting with mailing-lists. So how do I reconcile the
two?

One possibility I'm considering is forking the RForum webapp
(http://rubyforge.org/projects/rforum/) and shaping her up into a
"LibreForum".

Thoughts?

Re: RForum -> LibreForum

From:
Zed A. Shaw
Date:
2010-01-20 @ 18:40
On Wed, Jan 20, 2010 at 01:05:52PM -0500, Trans wrote:
> So I like Librelist, but I also like using a web-interface instead of
> my inbox for interacting with mailing-lists. So how do I reconcile the
> two?
> 
> One possibility I'm considering is forking the RForum webapp
> (http://rubyforge.org/projects/rforum/) and shaping her up into a
> "LibreForum".

Feel free to do that on your own servers, but I don't plan on trying to
support a web forum system at all.  They're difficult to run and scale,
open the door to even more spammers and trolls, and aren't necessary for
open source projects.

Now, if you wanted to work on better web archives I'd be very interested.
That's some which is much more useful than forum software.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: RForum -> LibreForum

From:
Trans
Date:
2010-01-20 @ 22:10
On Wed, Jan 20, 2010 at 1:40 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> l free to do that on your own servers, but I don't plan on trying to
> support a web forum system at all.  They're difficult to run and scale,
> open the door to even more spammers and trolls, and aren't necessary for
> open source projects.

I can host it myself, no problem.

> Now, if you wanted to work on better web archives I'd be very interested.
> That's some which is much more useful than forum software.

I am interested in an archive. But to me the difference between a good
archive and a web-interface to the list is a web form and a mail
library to send the post. I don't mean to let anonymous posts, and on
top of that I don't mind additional measures like captcha or login.

Since web archives are the first step toward a web-interface, I be
willing to work on it.

Re: RForum -> LibreForum

From:
Mike Dalessio
Date:
2010-01-22 @ 13:36
2010/1/20 Trans <transfire@gmail.com>

> On Wed, Jan 20, 2010 at 1:40 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> > l free to do that on your own servers, but I don't plan on trying to
> > support a web forum system at all.  They're difficult to run and scale,
> > open the door to even more spammers and trolls, and aren't necessary for
> > open source projects.
>
> I can host it myself, no problem.
>
> > Now, if you wanted to work on better web archives I'd be very interested.
> > That's some which is much more useful than forum software.
>
> I am interested in an archive. But to me the difference between a good
> archive and a web-interface to the list is a web form and a mail
> library to send the post. I don't mean to let anonymous posts, and on
> top of that I don't mind additional measures like captcha or login.
>
> Since web archives are the first step toward a web-interface, I be
> willing to work on it.
>

I'm willing to work on a web archive as well. I don't have a tremendous
amount of time to dedicate, but it's a pain point for me that I want to make
it go away.

Trans, feel free to ping me off list if there's anything I can help you
with.

Re: RForum -> LibreForum

From:
Mauricio Pasquier
Date:
2010-01-22 @ 16:31
On Fri, Jan 22, 2010 at 10:36, Mike Dalessio <mike.dalessio@gmail.com> wrote:
>
>
> 2010/1/20 Trans <transfire@gmail.com>
>> Since web archives are the first step toward a web-interface, I be
>> willing to work on it.
>
> I'm willing to work on a web archive as well. I don't have a tremendous
> amount of time to dedicate, but it's a pain point for me that I want to make
> it go away.
>
> Trans, feel free to ping me off list if there's anything I can help you
> with.
>

Idem here.

Re: RForum -> LibreForum

From:
Trans
Date:
2010-01-22 @ 16:32
On Fri, Jan 22, 2010 at 8:36 AM, Mike Dalessio <mike.dalessio@gmail.com> wrote:

> I'm willing to work on a web archive as well. I don't have a tremendous
> amount of time to dedicate, but it's a pain point for me that I want to make
> it go away.

The archive interface should be easy to knock out in a day or two. It
might be possible to do it all on the front end with jQuery.

As for threading, I figure it's good enough to go by subject. If the
subject changes it's a a new thread.

> Trans, feel free to ping me off list if there's anything I can help you
> with.

Ok. I started a github repo for it. (Sorry, not familiar enough with
Fossil at this point. We can move it to Fossil later if Zed deems the
results worthy of hosting.)

    http://github.com/trans/librelive

My only concern is the frequency of rsyncs. To keep the archive fresh
means rsyncing fairly frequently. rsync will do for now, but I'm
guessing the best solution will be giving the app it's own email
address by which it can receive new posts and add them to the archive
itself. That will keep it up to date in real-time. Then every so often
use rsync to make sure all posts were in fact received. That will
require a little server-side code, perhaps Sinatra (or do we stick to
Python, and if so is there something like Sinatra for Python we can
use)?

Re: RForum -> LibreForum

From:
Zed A. Shaw
Date:
2010-01-22 @ 17:12
On Fri, Jan 22, 2010 at 11:32:51AM -0500, Trans wrote:
> The archive interface should be easy to knock out in a day or two. It
> might be possible to do it all on the front end with jQuery.
> 
> As for threading, I figure it's good enough to go by subject. If the
> subject changes it's a a new thread.

A few features if you want people to adopt this for their projects so
they can host their own archives:

* Make it easily installable on a unix system.
* Static files only so you only need to run a web server.
* Crontab friendly so they can run it periodically.

> Ok. I started a github repo for it. (Sorry, not familiar enough with
> Fossil at this point. We can move it to Fossil later if Zed deems the
> results worthy of hosting.)
> 
>     http://github.com/trans/librelive

Whatever works.

> My only concern is the frequency of rsyncs. To keep the archive fresh
> means rsyncing fairly frequently.

That's entirely not necessary.  It's not a MUA, it's an archive.  30
minutes is more than frequently enough for an archive.

> That will keep it up to date in real-time. Then every so often
> use rsync to make sure all posts were in fact received.

At that point you might as well install librelist yourself, so again, I
think it's out of scope for an archive system.

Resist the tempation to make the archiver some grand vision of
web->email synchronization.  Keep it simple.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: RForum -> LibreForum

From:
Billy Gray
Date:
2010-01-22 @ 17:53
I agree, what I personally want is a sanitized archive on the web so that
one can easily search through it, have it indexed,  and features like 'show
next/prev message' (flat style) and 'show next message in thread' would be
clutch.

On Fri, Jan 22, 2010 at 12:12 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:

> On Fri, Jan 22, 2010 at 11:32:51AM -0500, Trans wrote:
> > The archive interface should be easy to knock out in a day or two. It
> > might be possible to do it all on the front end with jQuery.
> >
> > As for threading, I figure it's good enough to go by subject. If the
> > subject changes it's a a new thread.
>
> A few features if you want people to adopt this for their projects so
> they can host their own archives:
>
> * Make it easily installable on a unix system.
> * Static files only so you only need to run a web server.
> * Crontab friendly so they can run it periodically.
>
> > Ok. I started a github repo for it. (Sorry, not familiar enough with
> > Fossil at this point. We can move it to Fossil later if Zed deems the
> > results worthy of hosting.)
> >
> >     http://github.com/trans/librelive
>
> Whatever works.
>
> > My only concern is the frequency of rsyncs. To keep the archive fresh
> > means rsyncing fairly frequently.
>
> That's entirely not necessary.  It's not a MUA, it's an archive.  30
> minutes is more than frequently enough for an archive.
>
> > That will keep it up to date in real-time. Then every so often
> > use rsync to make sure all posts were in fact received.
>
> At that point you might as well install librelist yourself, so again, I
> think it's out of scope for an archive system.
>
> Resist the tempation to make the archiver some grand vision of
> web->email synchronization.  Keep it simple.
>
> --
> Zed A. Shaw
> http://zedshaw.com/
>



-- 
Billy Gray
http://zetetic.net

Re: RForum -> LibreForum

From:
Eric Wong
Date:
2010-01-22 @ 23:16
Trans <transfire@gmail.com> wrote:
> On Fri, Jan 22, 2010 at 8:36 AM, Mike Dalessio <mike.dalessio@gmail.com> wrote:
> 
> > I'm willing to work on a web archive as well. I don't have a tremendous
> > amount of time to dedicate, but it's a pain point for me that I want to make
> > it go away.
> 
> The archive interface should be easy to knock out in a day or two. It
> might be possible to do it all on the front end with jQuery.

Hi,

Any chance this can made search engine-friendly?  I've often found
search engine results pointing to old mailing list archives a great
resource.

> As for threading, I figure it's good enough to go by subject. If the
> subject changes it's a a new thread.
> 
> > Trans, feel free to ping me off list if there's anything I can help you
> > with.
> 
> Ok. I started a github repo for it. (Sorry, not familiar enough with
> Fossil at this point. We can move it to Fossil later if Zed deems the
> results worthy of hosting.)
> 
>     http://github.com/trans/librelive
> 
> My only concern is the frequency of rsyncs. To keep the archive fresh
> means rsyncing fairly frequently. rsync will do for now, but I'm
> guessing the best solution will be giving the app it's own email
> address by which it can receive new posts and add them to the archive
> itself. That will keep it up to date in real-time. Then every so often
> use rsync to make sure all posts were in fact received. That will
> require a little server-side code, perhaps Sinatra (or do we stick to
> Python, and if so is there something like Sinatra for Python we can
> use)?

I actually made some shell scripts and makefile to drive mbox mirrors
way back when Librelist started last year.  They try to place minimal
load on the rsync server by only mirroring messages for a given day.

I like the mbox format for long-term archives because you can fetch them
in one request and import/open them from most mail clients very easily
(and automatically get threading).

You can read more about it here:
  http://bogomips.org/librelist-mirror/

You can get the mbox file for this month, this list here:

  http://bogomips.org/librelist-mirror/meta/2010/01.gz

I suppose I should ungzip the mboxes by default to make life easier
for search engine crawlers, too.

-- 
Eric Wong

Re: RForum -> LibreForum

From:
Trans
Date:
2010-01-23 @ 16:44
I just pushed some code. Not much but it produces some results. If you
want to try it then clone the repo and run

    $ script/download meta
    $ script/build

And configure your local web server to serve up the site/ directory.

That's the beginnings of a static solution. Feel free to jump in
--there are still some very basic things to fix yet. And yes the
static archive will be quite indexable by search engines.

I am also working on a dynamic solution use Sinatra. Some of the code
base is going to be the same for both static and dynamic versions so I
decided to just support both approaches in a single application. When
all is said and done you will be able to download this application and
either generate a static site or fire-up the a dynamic server. Your
choice.

My apologize that it's in Ruby and not Python (to match LibreList's
language choice), I have no Python experience.

Re: RForum -> LibreForum

From:
Zed A. Shaw
Date:
2010-01-23 @ 17:13
On Sat, Jan 23, 2010 at 11:44:55AM -0500, Trans wrote:
> I just pushed some code. Not much but it produces some results. If you
> want to try it then clone the repo and run

Where's it at again?

> My apologize that it's in Ruby and not Python (to match LibreList's
> language choice), I have no Python experience.

Trust me, librelist is looking like it will motivate a large number of
archivers in multiple lanugages.  Ruby's just fine.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: RForum -> LibreForum

From:
Trans
Date:
2010-01-24 @ 00:44
On Sat, Jan 23, 2010 at 12:13 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> On Sat, Jan 23, 2010 at 11:44:55AM -0500, Trans wrote:
>> I just pushed some code. Not much but it produces some results. If you
>> want to try it then clone the repo and run
>
> Where's it at again?

http://transcode.github.com/librelive

>> My apologize that it's in Ruby and not Python (to match LibreList's
>> language choice), I have no Python experience.
>
> Trust me, librelist is looking like it will motivate a large number of
> archivers in multiple lanugages.  Ruby's just fine.

Cool Beans. Thanks.

Re: RForum -> LibreForum

From:
Trans
Date:
2010-01-24 @ 01:01
btw, I've always thought ruby-talks' archives were pretty good,
especially the threaded split-pane.

    http://blade.nagaokaut.ac.jp/ruby/ruby-talk/index.shtml
    http://blade.nagaokaut.ac.jp/ruby/ruby-talk/1-200.shtml
    
http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-talk/50?1-208+split-mode-vertical

Re: RForum -> LibreForum

From:
Billy Gray
Date:
2010-01-24 @ 20:38
Despite my utter loathing of frames, the vertical split mode is pretty
handy. However, I'd actually say this is even better:

http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-talk/1026?1009-1214

FWIW, I've always hated the crappy and non-descriptive navigation links of
the ruby-talk archive. They're a nuisance to look at and to figure out.

From the cheap seats,
Billy

On Sat, Jan 23, 2010 at 8:01 PM, Trans <transfire@gmail.com> wrote:

> btw, I've always thought ruby-talks' archives were pretty good,
> especially the threaded split-pane.
>
>    http://blade.nagaokaut.ac.jp/ruby/ruby-talk/index.shtml
>    http://blade.nagaokaut.ac.jp/ruby/ruby-talk/1-200.shtml
>
> 
http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-talk/50?1-208+split-mode-vertical
>



-- 
Billy Gray
http://zetetic.net

Re: RForum -> LibreForum

From:
Trans
Date:
2010-01-25 @ 03:40
On Sun, Jan 24, 2010 at 3:38 PM, Billy Gray <wgray@zetetic.net> wrote:
> Despite my utter loathing of frames, the vertical split mode is pretty
> handy. However, I'd actually say this is even better:
> http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-talk/1026?1009-1214

I can use divs and ajax loading instead of frames. Which do people prefer?

In either case looks like I'm headed toward a pluggable template
system, so ultimately there will be multiple options for layout.

> FWIW, I've always hated the crappy and non-descriptive navigation links of
> the ruby-talk archive. They're a nuisance to look at and to figure out.

I agree with you there. Won't be doing that.

~trans

Re: RForum -> LibreForum

From:
ssteinerX@gmail.com
Date:
2010-01-25 @ 03:42
On Jan 24, 2010, at 10:40 PM, Trans wrote:

> On Sun, Jan 24, 2010 at 3:38 PM, Billy Gray <wgray@zetetic.net> wrote:
>> Despite my utter loathing of frames, the vertical split mode is pretty
>> handy. However, I'd actually say this is even better:
>> http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-talk/1026?1009-1214
> 
> I can use divs and ajax loading instead of frames. Which do people prefer?

Frames suck.

S

Re: RForum -> LibreForum

From:
Zed A. Shaw
Date:
2010-01-25 @ 03:43
On Sun, Jan 24, 2010 at 10:40:10PM -0500, Trans wrote:
> I can use divs and ajax loading instead of frames. Which do people prefer?
> 
> In either case looks like I'm headed toward a pluggable template
> system, so ultimately there will be multiple options for layout.

That works, down side though is it's totally unsearchable by google.  I
think people would want it to be an HTML page.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: RForum -> LibreForum

From:
ssteinerX@gmail.com
Date:
2010-01-25 @ 03:46
On Jan 24, 2010, at 10:43 PM, Zed A. Shaw wrote:

> On Sun, Jan 24, 2010 at 10:40:10PM -0500, Trans wrote:
>> I can use divs and ajax loading instead of frames. Which do people prefer?
>> 
>> In either case looks like I'm headed toward a pluggable template
>> system, so ultimately there will be multiple options for layout.
> 
> That works, down side though is it's totally unsearchable by google.  

Depends on how it's done.  

If the div is there, just not displayed via javascript style mutilation, 
Google seems to pick it up just fine.

S

Re: RForum -> LibreForum

From:
Harry Vangberg
Date:
2010-01-20 @ 18:10
I have a tiny Sinatra app that uses the JSON API to expose the list
archives as static HTML. My only issue is getting them threaded - is
there any way Librelist tells me the individual relationship of
messages, Zed?

2010/1/20 Trans <transfire@gmail.com>:
> So I like Librelist, but I also like using a web-interface instead of
> my inbox for interacting with mailing-lists. So how do I reconcile the
> two?
>
> One possibility I'm considering is forking the RForum webapp
> (http://rubyforge.org/projects/rforum/) and shaping her up into a
> "LibreForum".
>
> Thoughts?
>



-- 
Harry Vangberg  <harry@vangberg.name>  http://harry.vangberg.name

Re: RForum -> LibreForum

From:
Zed A. Shaw
Date:
2010-01-20 @ 18:56
On Wed, Jan 20, 2010 at 07:10:59PM +0100, Harry Vangberg wrote:
> I have a tiny Sinatra app that uses the JSON API to expose the list
> archives as static HTML. My only issue is getting them threaded - is
> there any way Librelist tells me the individual relationship of
> messages, Zed?

Alright, I stopped long enough to get it working.  Try this:

rsync -azv librelist.com::json/meta meta

That syncs down an archive as all json files to your disk.  You can just
serve them up straight as files or parse them however you need.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: RForum -> LibreForum

From:
Zed A. Shaw
Date:
2010-01-20 @ 18:47
On Wed, Jan 20, 2010 at 07:10:59PM +0100, Harry Vangberg wrote:
> I have a tiny Sinatra app that uses the JSON API to expose the list
> archives as static HTML. My only issue is getting them threaded - is
> there any way Librelist tells me the individual relationship of
> messages, Zed?

I'm going to get the json on the rsync server too (soon as I find time)
which should make it even easier to do this.  No need to troll it
with HTTP just rsync down the JSON files and go to town.

Regarding the threading, take a look at the headers:


http://librelist.com/archives/meta/2010/01/20/json/1264011052.M719497P21396Q2737.09c5769d5b9f3d575cefc2ccb51877ec.json

The "References" and "Message-Id" headers are what you want.  Warning
though, getting threading right with those is a mofo and a half.  It's
doable but there's all sorts of problems with it.  Check out:

http://www.jwz.org/doc/draft-ietf-imapext-thread-12.txt

and

http://www.jwz.org/doc/threading.html

To get started.

Also, you should know that threading is harder for people who use
archives for information finding.  Typically someone writes threaded
archives as if they are reading the conversation, but that's usually
better done in your own mail client.

What archives usually get used for is finding the answers to past
questions, especially in open source projects.  When the messages are
threaded it's really difficult to find the original questions and then
the possible answers.  When they're flat it's really easy.


-- 
Zed A. Shaw
http://zedshaw.com/

Re: RForum -> LibreForum

From:
Trans
Date:
2010-01-20 @ 21:06
On Wed, Jan 20, 2010 at 1:47 PM, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
> Also, you should know that threading is harder for people who use
> archives for information finding.  Typically someone writes threaded
> archives as if they are reading the conversation, but that's usually
> better done in your own mail client.
>
> What archives usually get used for is finding the answers to past
> questions, especially in open source projects.  When the messages are
> threaded it's really difficult to find the original questions and then
> the possible answers.  When they're flat it's really easy.

I find both views useful. Flat is good for search, but once I find
what I'm looking for I switch threaded mode (if I can) to read the
entire conversation.